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NOTICE 


This  report  is  one  of  several  cost  benefit  analysis  efforts  in  the  area  of  human  effectiveness 
research  sponsored  by  AFRL/HE.  Other  topics  include  Bioacoustic  Protection,  Distributed 
Mission  Training  (DMT),  and  a  state-of-the-art  report  (SOAR)  that  outlines  the  general 
methodology  to  conduct  effective  cost  benefit  analyses  on  human  centered  technologies. 
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Three-dimensional 

ACC 

Air  Combat  Command 
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Air  Combat  Capability  Enhancement  Suite 

AFSC 
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ASC 
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DoD 

Department  of  Defense 
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IV 


EXECUTIVE  SUMMARY 


A.  Overview 

At  the  request  of  the  Air  Force  Research  Laboratory,  Fluman  Systems  lAC  assessed  the 
costs  and  relative  benefits  of  three  select  alternatives  for  improving  accessibility  of 
anthropometric  data  for  military  and  civilian  use.  The  three  alternatives  compared  in  our  analysis 
were: 

•  Status  Quo, 

•  Searchable  Database,  and 

•  Searchable  Database  with  Expert  Interface 

The  primary  goal  of  the  alternatives  assessed  was  to  develop  an  anthropometric  size, 
shape,  fit  and  accommodation  information  system  for  efficient  and  effective  use  by  designers, 
manufacturers,  buyers,  and  decision  makers  across  a  number  of  domains. 

B.  Study  Methodology 

1.  Costs 

Costs  for  the  alternatives  analyzed  in  this  CBA  were  estimated  using  a  variety  of 
methodologies.  Methodologies  used  include  actual  costs,  expert  opinions  and  estimating  models. 
A  summary  of  each  alternative's  groundrules  and  assumptions  can  be  found  in  Section  2.4.  Cost 
estimating  methodology  matrices  and  detailed  worksheets  for  each  are  included  in  Appendix  A. 
Table  1  summarizes  the  total  cost,  in  both  constant  (FY02)  dollars  and  then  year  (inflated) 
dollars,  by  program  phase  for  each  of  the  alternatives. 


Table  1.  Summary  of  Alternative  Costs 


Constant  FY02  $000 

Alternative  #1 

Alternative  #2 

Alternative  #3 

System  Acquisition 

$0 

$2,136.4 

$6,072.0 

System  Implementation 

$66.23 

$602.2 

$354.0 

Operation  and  Support 

$4,918.50 

$4,971.0 

$7,024.7 

Total 

$4,984.73 

$7,736.6 

$13,450.7 

Then  Year  $000 

Alternative  #1 

Alternative  #2 

Alternative  #3 

System  Acquisition 

$0 

$2,294.0 

$6,334.4 

System  Implementation 

$70.33 

$629.8 

$371.4 

Operation  and  Support 

$5,222.59 

$5,268.5 

$7,497.5 

Total 

$5,292.92 

$8,147.3 

$14,203.3 

2.  Benefits 

Benefits  were  assessed  in  several  ways  as  well.  Many  benefits  were  clear  from 
information  gathered  in  the  literature  search.  Additional  benefits  were  identified  through  an 
exhaustive  coordination  with  subject  matter  experts  and  users  who  possess  real  world  experience 
with  the  anthropometric  information  challenges  being  analyzed.  Table  2  summarizes  the  agreed- 
upon  benefits. 
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Table  2.  Benefits  Definitions 


Benefit  Name 

Benefit  Definition 

Accessibility 

System  impact  on  ease  of  accessing  anthropometric  information;  the 
number  and  type  of  individuals  that  can  locate  the  info. 

Accuracy 

System  impact  on  currency,  correctness,  and  direct  use  of 
fit/accommodation  information.  Information  provided  is  the  correct 
solution  for  the  task  at  hand. 

Efficiency 

System  impact  on  speed  of  finding  anthropometric 
fit/accommodation  information.  A  more  efficient  system  will  require 
less  time  invested. 

Scope 

System  impact  on  sufficiency  of  anthropometric  fit/accommodation 
information  provided  to  user.  Information  provided  is  in-depth 
enough  to  meet  user's  needs. 

Usability 

Ease  of  system  use  in  accessing  anthropometric  fit/accommodation 
information.  Eevel  of  effort  required  of  designer  to  find  information. 

A  highly  useable  system  requires  little  effort. 

The  assessment  of  these  benefits  was  documented  using  a  decision  support  software  tool, 
Expert  Choice.  The  tool  was  used  to  document  the  detailed  assessment  process  and  numerically 
rank  the  three  alternatives.  Expert  Choice  leads  the  decision-maker  through  a  series  of  judgments 
between  the  alternatives  and  then  between  the  benefits.  The  tool  then  combines  all  the  priorities 
to  arrive  at  an  overall  ranking  of  the  alternatives.  The  resulting  benefits  analysis  showed  that, 
with  respect  to  the  prescribed  goal,  alternative  #3,  with  a  score  of  63.3,  best  satisfies  the 
requirements.  Alternative  #2  is  the  next  best  alternative,  scoring  29.7.  The  least  beneficial 
alternative,  alternative  #1,  scored  a  7.0. 

3.  Cost/Benefit  Ratio 

The  cost/benefit  ratio  (CBR)  was  used  as  a  method  of  combining  the  costs  with  the 
benefits  to  establish  an  overall  conclusion  regarding  the  investment  alternatives.  Combining 
costs  and  benefits  determines  the  true  value  of  each  alternative.  The  CBR  represents  the  cost  per 
unit  of  benefit.  Thus,  a  lower  cost/benefit  ratio  is  preferred. 


Table  3.  Alternative  Cost  Benefit  Ratios  and  Rankings 


Alternative 

Cost 

(Constant  FY02  $000) 

Benefit 

Score 

Cost/Benefit 

Ratio 

Ranking 

#1  Status  Quo 

$4,984.73 

7.0 

712.1 

3 

#2  Searchable  D-Base 

$7,736.6 

29.7 

260.5 

2 

#3  Searchable  D-Base  w/  Expert  Int. 

$13,450.7 

63.3 

212.5 

1 

Based  on  this  data,  alternative  #3  was  the  most  cost-effective.  The  analysis  indicates  it 
provided  the  greatest  benefit  for  the  money,  with  respect  to  the  stated  goal.  Alternative  #1  ranked 
the  least  cost-effective  and  alternative  #2  fell  in  the  middle. 
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4,  Sensitivity  Analysis 

The  sensitivity  analysis  focused  on  two  areas.  First  was  the  dependence  on  expert 
opinion  to  estimate  the  software  development  effort  common  to  both  alternatives  #2  and  #3. 

Since  this  portion  of  the  estimate  was  the  same  for  both  alternatives  ($1.9M),  variations  in  the 
input  variables  may  change  the  total  cost,  but  would  not  impact  the  comparative  analysis  between 
these  two  alternatives. 

The  second  area  of  sensitivity  was  the  method  used  to  estimate  the  expert  interface 
function  unique  to  alternative  #3..  The  estimate  for  this  portion  of  alternative  #3  was  based 
largely  on  information  obtained  from  anthropometric  experts  envisioning  how  an  expert  system 
could  deliver  accurate  anthropometric  information.  Flowever,  even  if  this  particular  cost  element 
is  overstated  by  as  much  as  50  percent,  the  total  cost  of  alternative  #3  would  still  exceed  the  cost 
of  alternative  #2. 

5.  Risk  Analysis 

From  a  cost  perspective,  two  areas  were  addressed  for  their  potential  risk  to  the  overall 
analysis.  These  are  the  application  of  expert  interface  technology  in  the  area  of  anthropometry 
and  the  impact  of  expert  opinion  as  a  primary  data  source. 

A  significant  portion  of  the  total  cost  of  alternatives  #2  and  #3  related  to  the  cost 
element.  Software  Investment.  Specifically,  the  introduction  of  an  expert  interface  function  into 
alternative  #3  results  in  an  increase  of  $3.8M  over  alternative  #2  for  this  element  alone.  This 
makes  the  accuracy  of  the  expert  interface  estimate  crucial  to  the  credibility  of  the  overall 
rankings.  As  a  result,  this  area  may  require  additional  research  addressing  the  specific  impact  on 
the  cost  of  the  alternatives  at  that  time. 

The  second  area  of  cost  risk  relates  to  the  use  of  expert  opinion  as  the  primary  source  of 
cost  data  for  the  three  air  combat  capability  enhancement  suite  (ACCES)  alternatives.  While 
some  of  the  elements  estimated  using  this  source  were  based  on  known  values  (e.g.,  labor  costs, 
hardware  and  software  budgets,  etc),  others  required  the  experts  to  envision  a  computerized 
capability  that  is,  as  of  yet,  unproven.  As  a  result,  this  data,  while  the  best  available,  is  not  based 
on  documented  information. 

There  were  also  two  areas  of  risk  identified  with  the  benefits  assessment.  The  first  area 
was  the  absence  of  practical  experience  with  the  actual  alternatives  by  the  experts  assessing  one 
or  more  of  the  alternatives.  The  risk  emerged  in  alternative  #3,  the  searchable  database  with  an 
expert  interface.  The  potential  variability  was  that  the  assessment  of  benefits  related  to  this 
solution  had  to  rely  on  the  alternative  as  it  was  defined  and  its  expected  successful  application  in 
the  anthropometric  community.  There  was  no  actual  entity  to  evaluate.  As  a  result,  there  is  risk 
that  the  technology  may  not  work  as  expected  when  fully  developed  and  used  in  this 
environment. 

The  second  risk  area  related  to  the  benefit  assessment  was  that  no  novice  users  were 
included  in  the  pool  of  experts  who  participated  in  the  Expert  Choice  session.  The  group 
consisted  of  individuals  that  had  extensive  experience  in  the  area  of  interest  and  was  made  up  of 
both  government  and  industry  experts.  However,  if  a  new  Expert  Choice  group  was  assembled,  it 
is  possible  that  the  alternatives  may  be  ranked  in  a  different  way,  based  on  their  own  personal 
experiences  and  opinions. 
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6.  Conclusions 


This  study  evaluated  the  costs  and  benefits  of  three  anthropometric  information  system 
alternatives.  Using  the  cost  estimate  and  benefit  analysis  developed  for  each  alternative,  Human 
Systems  lAC  calculated  a  cost/benefit  ratio  for  each  of  the  alternatives. 

The  most  effective  alternative  with  respect  to  the  stated  goals  was  alternative  #3  the 
searchable  database  with  expert  interface,  (CBR  =  212.5).  The  second  most  desirable  choice  was 
alternative  #2,  the  searchable  database  (CBR  =  260.5).  Based  on  the  evaluated  cost  and  benefits 
the  least  cost-effective  solution  for  providing  anthropometric  information  was  alternative  #1,  the 
status  quo  (CBR  =  712.1). 
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1.  INTRODUCTION 


I.I  BACKGROUND 

Designing  for  the  user,  be  it  accommodation  in  a  crew  station/cockpit  or  sizing  of  personal 
equipment  and  apparel,  is  often  accomplished  using  outdated  information  or  arbitrary,  unrelated 
criteria.  The  result  is  poorly  sized  and  inefficiently  stocked  equipment.  Use  of  inaccurate  data 
also  results  in  errors  in  the  selection  process.  It  decreases  the  number  of  candidates  available  by 
eliminating  individuals  that  are  physically  capable  of  working  in  a  particular  cockpit/crew  station, 
while  including  some  that  are  not.  There  must  be  a  way  for  designers  of  crew  stations  and 
equipment  to  design  for  the  user  and  provide  for  the  selection  of  appropriate  users,  based  on 
effective  application  of  anthropometric  data. 

When  developing  and  designing  military  equipment  for  use  by  human  operators,  it  is 
important  to  design  for  the  user.  As  the  modem  battlefield  changes,  so  do  the  needs  of  designers. 
For  instance,  new  roles  for  women  in  combat  require  body  size  data  for  a  potential  population  of 
people,  some  of  whom  are  not  currently  in  the  military.  For  example,  the  new  U.S.  Air  Force  T- 
6  II  trainer  was  designed  to  accommodate  pilots  with  statures  ranging  from  4T0"  to  6'5" 
characterized  by  various  body  segment  proportions  in  an  effort  to  broaden  the  pool  of  potential 
pilots  and  meet  international  needs.  This  new  trainer,  along  with  the  F-22  and  Joint  Strike 
Fighter  (JSF),  employed  manikin  sizes  defined  by  the  multivariate  accommodation  method  rather 
than  percentile  method. 

Unfortunately,  anthropometric  dimensions  are  not  perfectly  correlated  with  one  another. 
For  example,  people  with  long  legs  do  not  necessarily  have  long  torsos.  This  forces  decision 
makers  to  consider  the  relationship  between  specific  body  dimensions  in  the  design  of  systems 
based  on  combinations  of  dimensions,  the  intended  use  and  user  population.  For  instance,  by 
designing  for  the  5*  to  95*  percentiles  on  more  than  one  dimension,  the  design  will  exclude 
significantly  more  individuals  than  the  implied  10  percent.  In  fact,  Bittner  (1974)  showed  in  one 
study  that  applying  5*  and  95*  limits  on  each  of  13  dimensions  actually  excluded  52  percent  of 
the  potential  user  population!  Bottom  line  is  that  it  is  essential  to  consider  the  body  dimensions 
important  to  the  design,  the  population  that  will  use  the  equipment,  and  resources  required  to 
provide  the  information  needed. 

When  poor  or  incorrect  anthropometric  and  accommodation  information  is  used,  the 
result  can  be  a  decrease  in  safety.  For  instance,  in  nine  percent  of  aircraft  accidents  over  the  past 
10  years  body  size  was  listed  as  a  contributor  (Zehner,  2001).  Designing  for  varying  body  sizes 
should  help  decrease  the  number  of  accidents.  Poor  anthropometric  accommodations  also  result 
in  uncomfortable/poor  fitting  garments  and  cockpits  that  cannot  accept  all  aircrew  that  it  is 
intended  to.  The  result  from  the  use  of  incorrect  information  leads  to  the  deployment  of 
equipment  and  operators  that  simply  cannot  safely  fit  in  all  of  the  cockpits  they  are  supposed  to 
support. 

Another  result  of  incorrect  information  is  that  the  equipment  available  does  not  fit  and/or 
cannot  be  used  by  those  that  need  it.  Use  of  inaccurate  data  also  result  in  errors  in  the  selection 
process  as  they  decrease  the  number  of  candidates  available  by  eliminating  individuals  that  are 
physically  capable  of  working  in  a  cockpit/crew  station,  while  including  some  that  are  not.  This 
is  exemplified  by  Bittner  (1974),  as  described  above. 

There  must  be  a  way  for  designers  of  crew  stations  and  equipment  to  design  for  the  user 
and  provide  for  the  selection  of  appropriate  users,  based  on  effective  application  of 
anthropometric  data.  The  effort  to  develop  the  air  combat  capability  enhancement  suite  (ACCES) 
will  help  decision  makers  make  educated  judgments  about  accommodation  information  system 
for  use  by  decision-makers  across  a  number  of  domains.  The  goal  will  be: 
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•  A  data  base  of  current  forces  and  the  best  equipment  size,  adjustments,  etc.  for  usability, 
fit  and  safety; 

•  Information  on  areas  of  improvement  for  more  cost-effective  fit,  assignment  entry,  fitness 
standards,  and  purchasing;  and 

•  The  best  mix  of  stocked/special  order/custom  personal  equipment  sizes  for  rapid  and 
cost-effective  deployment. 


1.2  PURPOSE  OF  CBA 

In  general,  a  cost  benefit  analysis  (CBA)  is  a  systematic  method  of  assessing  the 
economic  desirability  of  selected  investment  alternatives  based  on  their  costs  and  benefits, 
respectively.  The  objective  of  this  assessment  is  to  identify  one  alternative  that  is  preferable  to  all 
of  the  others. 

The  specific  objective  of  this  CBA  project  was  to  evaluate  the  economic  desirability  of 
three  proposed  anthropometric  information  systems  for  use  by  experts  and  novices,  based  on  their 
respective  cost  and  benefits.  The  alternative  eventually  implemented  should  provide  appropriate 
and  accurate  anthropometric  information  to  allow  designers  to  provide  for  the  selection  of 
appropriate  users.  The  alternative  must  also  provide  understandable,  adaptable  anthropometry 
and  accommodation  information  to  the  user.  Lastly,  the  alternative  must  incorporate  previously 
gathered  information  and  build  upon  research  in  3-D  shape  that  has  already  started. 

1.3  OVERVIEW  OF  APPROACH 

As  stated  above,  a  CBA  is  a  systematic  method  employed  to  make  rational  decisions  on 
possible  alternative  systems  by  comparing  selected  alternatives  based  on  their  estimated  costs  and 
evaluated  benefits.  Conducting  a  CBA  assists  in  the  allocation  of  scarce  resources  by  probing 
each  investment  alternative  so  that  all  questions  relevant  to  an  appropriate  decision  are  answered. 
The  overall  objective  and  alternatives  are  clearly  defined;  costs  and  benefits  are  fully  presented; 
and  important  assumptions,  factors  and  judgments  are  highlighted.  Outlined  in  the  following 
sections,  are  the  eight  steps  used  by  Human  Systems  lAC  in  conducting  a  CBA.  They  are: 

1 .  Assess  Current  Environment 

2.  Perform  Gap  Analysis 

3.  Identify  Investment  Alternatives 

4.  Estimate  Cost 

5.  Perform  Sensitivity  Analysis 

6.  Characterize  and  Evaluate  Benefits 

7.  Determine  Net  Value  of  Each  Alternative 

8.  Perform  Risk  Analysis 

Once  ground  rules  and  assumptions  are  established,  a  cost  estimate  is  prepared  for  each 
alternative  and  benefits  are  evaluated.  Each  estimate  is  then  normalized  to  a  constant  year  for 
cost  comparison  purposes.  Analysis  of  the  benefits  is  accomplished  using  decision  support 
software  (Expert  Choice)  that  enables  Human  Systems  lAC  to  leverage  the  expertise  of  key 
stakeholders  and  experts  in  the  field. 
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2.  COST  BENEFIT  ANALYSIS 


2.1  CURRENT  ENVIRONMENT 

2.1. 1  Process 

In  evaluating  the  current  environment  Human  Systems  lAC  investigated  the  types  of 
technologies  currently  being  applied.  The  need  for  a  material  solution  based  on  the  objectives 
was  also  addressed.  The  current  technology  in  anthropometric  information  was  surveyed  and 
thoroughly  examined  to  establish  a  baseline  for  the  analysis. 

When  assessing  the  current  environment,  Human  Systems  lAC  analysts  first  "got  smart" 
on  the  topic  at  hand.  This  began  with  an  in-house  literature  search  to  ascertain  the  general  nature 
of  the  topic.  In  this  case,  anthropometric  databases  were  researched.  Based  on  this  search,  a 
comprehensive  list  of  keywords,  catch  phrases,  subject  matter  experts  (SME),  and  example 
articles  was  compiled.  As  is  often  the  case,  the  customer  was  a  subject  matter  expert  in  the  field 
and  knew  a  great  deal  about  the  specific  topic  of  anthropometry.  Therefore,  this  search  strategy 
was  developed  in  close  coordination  with  customer  review  and  feedback.  Once  coordination  was 
received  on  the  search  strategy,  an  in-depth  literature  review  of  all  appropriate  databases  was 
conducted.  (See  Appendix  C  for  Search  Strategy.)  The  ACCES  search  resulted  in  the 
identification  of  literally  hundreds  of  citations.  These  citations  were  surveyed  for  the  most 
relevant  and  insightful  resources.  The  emerging  sources  became  the  backbone  literature  source  of 
information  used  throughout  the  CBA. 

Once  the  background  work  was  completed,  selected  documents  were  acquired  and  SMEs 
were  contacted.  At  this  point.  Human  Systems  lAC  gathered  all  available  documentation  and 
information  from  the  customer,  asking  numerous  questions  when  necessary.  The  developed 
information  was  then  combined  to  provide  an  accurate  perspective  of  the  current  system. 

2.1.2  Evaluation 

Currently,  the  most  effective  way  to  obtain  accurate  anthropometric  information  is  to 
contact  an  expert  consultant,  such  as  someone  at  AFRE/HE,  to  find  the  appropriate  data  and  put  it 
in  a  useful  form  for  the  particular  project  or  problem.  Once  contacted,  the  experts  search  the  most 
relevant  resources  available  and  provide  a  response.  The  consultant  may  or  may  not  have  the 
software  and  data  resources  readily  available  to  accomplish  this,  therefore  this  information  may 
have  to  be  separately  acquired.  In  addition  not  all  anthropometric  information  is  in  a  central 
location,  nor  can  it's  location  be  found  from  one  central  resource.  In  fact,  the  existence  of  the 
appropriate  information  and  the  existence  of  expertise  to  find  or  use  it  may  not  be  known  by  the 
user.  For  example,  a  manufacturer  of  an  oxygen  mask  may  not  know  there  is  a  3-D  face  data 
resource  available. 

This  approach  is  time  consuming  and  often  ineffective.  The  AFRE  experts  may  be  able 
to  provide  an  effective  solution  given  sufficient  time,  but  their  primary  job  is  to  develop  new 
technologies  and  transition  them,  not  to  execute  them  once  they  are  developed.  Since  it  is  not 
their  normal  job  to  serve  as  experts  on  acquisition  programs,  there  is  no  routine  mechanism  to  use 
AFRE  experts  in  the  acquisition  process.  As  a  result,  they  often  are  called  upon  to  help  very  late 
in  the  process  with  little  time  left  to  help.  Any  consulting  they  do  must  fit  into  their  normal 
duties.  This  further  reduces  their  ability  to  help  effectively  and  efficiently. 

At  the  time  of  this  report,  the  specific  goals  and  requirements  for  this  system  were 
defined  as  described  above,  but  remained  somewhat  vague.  No  mission  needs  statement  (MNS) 
or  operational  requirements  document  (ORD)  existed.  Nonetheless,  there  are  still  possible 
solutions  to  the  described  shortfalls  of  the  current  system.  Solutions  to  the  problem  of  accessing 
data  fall  within  two  categories:  material  solutions  or  doctrine  changes. 
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Material  solutions  would  include  improvements  to  computer  systems  associated  with 
information  storage  and  retrieval,  better  libraries,  etc.  Doctrine  changes  would  include  a 
completely  new  approach  for  handling  the  dissemination  of  useful,  accurate  information.  This 
solution  could  include  additional  training  or  new  information  handbooks.  An  additional 
consideration  to  help  solve  the  problem  would  be  to  remove  the  man  from  the  system.  For 
instance,  the  work  being  done  by  expert  individuals  could  be  conducted  via  an  "expert  system." 

2.2  GAP  ANALYSIS 

2.2.1  Process 

In  addressing  the  gap,  it  was  the  task  of  Human  Systems  lAC  to  identify  the  techniques 
and  technologies  that  work  well  and  the  underlying  reason  for  the  effectiveness.  This  step  of  the 
CBA  was  intended  to  illustrate  the  gap  between  what  is  and  what  is  supposed  to  be.  By 
understanding  the  desired  system's  requirements  and  the  effectiveness  of  the  current  technology 
(baseline)  in  terms  of  meeting  those  requirements,  it  was  determined  where  mission  needs  were 
not  being  met. 

2.2.2  Evaluation 

The  most  effective  systems  providing  anthropometric  information  included  those  that 
provided  complete  and  accurate  information  quickly.  For  instance,  consulting  a  textbook  may 
yield  information  quickly,  but  it  may  not  be  as  complete  or  up  to  date  and  accurate  as  a  less 
accessible  website.  Another  consideration  was  the  fact  that  current  anthropometric  information 
was  distributed  across  several  resources  and  databases,  making  it  difficult  to  acquire,  even  for 
experts. 

Technologies  that  worked  well  in  providing  anthropometric  information  vary  from  well- 
known  handbooks  to  cutting-edge  3-D  scanners.  Each  source  has  some  positives  and  negatives 
associated  with  it.  For  instance,  the  information  provided  in  the  Army  Anthropometric  Survey 
(1988)  is  relatively  easy  to  acquire  and  understand,  but  it  is  fast  becoming  obsolete.  On  the  other 
hand,  information  from  3 -dimensional  scans  is  up  to  date  and  very  complete,  but  the  sheer 
amount  of  data  makes  it  difficult  to  manage  to  reach  an  effective  solution.  Overall,  critical 
features  of  an  effective  solution  included  an  information  system  which  provided  accurate  and 
complete  information  quickly  and  easily.  Unfortunately,  no  system  existed  with  all  of  these 
attributes. 

When  an  effective  solution  was  compared  to  the  status  quo,  it  was  evident  there  was  a 
significant  gap  between  the  information  required  and  the  information  available. 

2.3  INVESTMENT  ALTERNATIVES 

2.3.1  Process 

Utilizing  information  from  assessment  of  the  current  environment  and  the  gap  analysis. 
Human  Systems  lAC  then  added  customer  input  and  the  opinions  of  technical  experts  to  identify 
potential  investment  alternatives  to  investigate. 

Alternatives  can  be  identified  in  several  ways.  In  general,  the  status  quo  (currently 
fielded  system)  is  one  of  the  alternatives  examined  as  it  establishes  a  baseline  for  comparison. 
Additional  alternatives  are  often  the  next  generation  of  technology  identified  in  the  literature 
search.  Human  Systems  lAC  analysts,  SMEs,  or  the  customer  identify  further  choices. 
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2.3.2  Evaluation 


The  specific  goal  of  the  technologies  under  scrutiny  was  to  develop  an  anthropometric 
size,  shape,  fit  and  accommodation  information  system  for  efficient  and  effective  use  by 
designers,  manufacturers,  buyers,  and  decision  makers  across  a  number  of  domains.  The  solution 
should  also  provide  understandable,  adaptable  anthropometric  and  accommodation  information  to 
the  user. 

It  was  established  during  the  assessment  of  the  current  environment  that  material  and 
doctrine  solutions  were  necessary.  Key  attributes  of  a  material  solution  included  technological 
improvements  in  the  area  of  data  storage  and  retrieval.  Another  important  aspect  of  the  material 
solution  would  be  improved  database  capabilities  and  enhanced  accessibility  to  information. 

Key  attributes  of  a  doctrine  change  would  include  a  fresh  look  at  the  use  of 
anthropometric  information  by  individuals.  It  would  provide  for  the  effective  delivery  of 
accurate  information  to  the  right  people  through  training  of  those  individuals  or  some  other 
method.  An  expert  interface  that  makes  key  decisions  for  the  user  could  help  alleviate  the  need 
for  experts  in  the  loop  or  additional  training. 

Human  Systems  lAC  identified  three  alternatives  to  analyze  with  respect  to  these 
features.  The  first  was  the  status  quo  information  system.  This  alternative  was  included  because 
it  provided  a  baseline  for  comparison.  The  second  investment  alternative  was  a  searchable 
database.  This  was  the  first  step  in  improving  information  accessibility  for  the  user.  The  third 
alternative  addressed  in  this  CBA  was  a  searchable  database  with  an  expert  interface.  This  final 
alternative  incorporated  all  of  the  requirements  and  used  the  latest  technology. 

2.3.2.1  Alternative  #1,  Status  Quo 

Described  fully  in  assessment  of  the  current  environment  (Section  2.1.2),  the  status  quo 
alternative  included  the  additional  resources  described  below: 

•  Printed  texts  with  summary  statistics  from  preset  surveys  such  as  mean,  standard 
deviation  and  percentiles.  Sample  may  not  be  representative  of  the  desired  population. 

•  Printed  standards  and  handbooks  such  as  MIL-STD-1472  and  MIL-HDBK-743. 

•  Journal  articles  and  human  factors  textbooks. 

•  One-dimensional  (traditional  tape  measure,  caliper  type)  data  by  survey  in  spreadsheets 
on  CD,  such  as  the  USAF  mini-survey. 

•  Collections  of  3-D  scans  in  original  scan  form  on  CD,  such  as  the  Dayton  Survey  and  the 
USAF  HGU-55/P  survey. 

2.3.2.2  Alternative  #2,  Searchable  Database 

This  alternative  consists  of  an  on-line  one-  and  three-dimensional  database  made  up  of  all 
available  Air  Force  anthropometric  resources  for  fit  and  accommodation,  with  sample  merging, 
weighting  and  segmenting  capability.  This  searchable  Oracle-based  database  will  bring  together 
previously  unincorporated  sources;  information  will  either  be  located  in  the  database  or  its 
location  can  be  identified  from  the  central  information  system.  For  example,  when  a  query  is 
made  to  the  database,  a  list  of  resources  is  generated.  From  this,  the  user  accesses  the  suggested 
resources,  builds  a  data  set  and  utilizes  the  statistics  package  to  formulate  an  answer  to  his/her 
anthropometric  question.  Sometimes  the  answer  will  tell  them  what  NEW  data  to  collect  and 
how  to  collect  it.  The  data  they  need  may  not  be  in  the  database  yet.  Also,  when  they  collect 
new  data  they  will  want  to  add  it  to  the  database  for  future  use. 

Because  using  the  information  still  requires  a  knowledgeable  expert  to  answer  the 
questions  in  many  cases,  training  is  a  key  element  of  this  alternative.  In  order  to  get  the  same 
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utility  from  this  software  as  asking  the  experts,  a  user  would  have  to  be  trained  to  ask  the  correct 
questions  and  make  appropriate  decisions  based  on  the  information  presented.  For  example,  if 
the  user  wanted  to  know  which  individuals  could  safely  fit  into  a  crew  station,  he/she  would  have 
had  to  know  which  measurements  were  important,  which  database  had  the  most  representative 
sample  of  those  measurements,  gather  the  information  from  the  sources  in  the  database,  make  any 
necessary  calculations  to  normalize  and  average  the  date,  and  then  make  a  decision  based  on  the 
available  information.  The  training  course  accompanying  this  alternative  will  guide  the  user 
through  the  information  sources  and  process  required  to  arrive  at  an  accurate  anthropometric 
solution. 

In  summary,  this  database  would  enable  an  expert  to  access  the  information  and  arrive  at 
solutions  quickly  and  easily,  thereby  increasing  their  effectiveness.  It  would  also  allow  non¬ 
experts  to  easily  gain  an  understanding  of  what  information  is  available.  The  searchable  database 
would  include  the  following  resources: 

•  Lessons  learned  and  fit  and  accommodation  maps, 

•  Human  System  Equipment  size  and  shape  information, 

•  Interfacing  3-D  visualization  and  statistical  analysis  tools,  including  multivariate  analysis 
software,  and 

•  Electronic  accommodation  or  fit  prediction  models. 

2.3.2.3  Alternative  #3,  Searchable  Database  with  Expert  Interface 

This  alternative  is  basically  the  same  as  alternative  two,  a  searchable  on-line  one-  and 
three-dimensional  database  made  up  of  all  available  Air  Force  anthropometry  resources  for  fit  and 
accommodation,  with  sample  merging,  weighting  and  segmenting  capability.  However,  this 
alternative  adds  an  expert  user  interface  in  the  form  of  standard  decision  templates  to  allow  non¬ 
experts  to  obtain  and  effectively  use  the  information  they  need  without  additional  training.  For 
example,  when  a  query  is  made  to  the  database,  a  list  of  resources  is  generated.  Some  of  these 
may  be  available  from  within  the  database,  and  some  sources  may  simply  list  the  location  they 
can  be  retrieved  from.  The  user  is  walked  through  a  series  of  questions  based  on  standard 
templates  to  simulate  various  anthropometric  scenarios.  Ideally,  no  complex  calculations  or 
expert  decisions  will  be  required.  The  user  would  simply  have  to  ask  a  question  such  as  "Which 
individuals  can  safely  fit  into  a  crew  station  with  x-dimensions?"  and  the  database  would  walk  the 
user  through  the  appropriate  steps  to  provide  that  answer.  The  searchable  database  with  expert 
interface  would  include  the  following  resources: 

•  An  interface  that  would  allow  a  novice  to  locate  the  appropriate  information  in  the  format 
needed,  by  simply  asking  a  question.  The  program  would  walk  them  through  the  answer 
to  questions  like: 

•  Would  this  candidate  be  able  to  safely  fly  USAF  fighter  aircraft?  Answer  would 
include  the  aircraft  that  could  be  safely  flown  as  well  as  any  areas  that  might  be  on 
the  borderline  and  by  how  much. 

•  What  percentage  of  the  population  would  be  able  to  fly  this  aircraft  if  this  change 
were  made?  The  interface  would  ask  the  novice  specific  questions  to  ascertain 
exactly  what  the  population  of  interest  might  be.  The  answer  would  include  the 
overall  population,  the  male  and  the  female  populations  and  the  minority  population 
percentages. 

•  What  should  the  requirements  be  for  a  new  oxygen  mask?  The  interface  would  ask 
the  novice  questions  to  ascertain  how  the  oxygen  mask  is  intended  to  fit  and  function, 
and  who  the  intended  users  would  be  using  examples  and  pictures  (3-D  if  available) 
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from  the  fit  history  files.  The  answer  would  include  the  statements  to  be  written  in 
the  specification,  and  the  data  on  the  cases  (3-D  if  necessary  and  available)  to  use  to 
accommodate  the  population. 


2.3.2.4  Global  Ground  Rules  and  Assumptions 

The  following  ground  rules  and  assumptions  were  used  in  estimating  the  costs  and 
evaluating  the  benefits  of  each  ACCES  alternative: 

•  Three  alternatives  for  retrieving  anthropometric  information  were  examined. 

•  Each  alternative  was  estimated  with  the  baseline  goal  of  providing  full  capability 
anthropometric  information  for  a  3 -year  period.  Program  schedules  differ  due  to  the 
varying  assumptions  for  acquisition  and  implementation  costs  for  each;  Costs  for  each 
alternative  begin  in  FY02. 

•  Costs  for  each  alternative  are  assumed  to  be  Air  Force  appropriation  3600,  Research, 
Development  Test  and  Evaluation  (RDT&E)  funds. 

•  All  detailed  costs  were  presented  in  both  constant  dollars  and  then  year  dollars  using 
standard  DoD  inflation  rates. 

•  Composite  burdened  labor  rates  were  used  for  all  civilian  and  military  labor. 

•  Groundrules  and  assumptions  specific  to  a  particular  alternative  were  summarized  with 
the  documentation  for  that  alternative. 

•  The  sources  of  information  for  all  three  alternatives  are  of  three  (3)  types:  1.  Hard  copy 
(regulations,  technical  reports  and  handbooks)  2.  Electronic  databases  and  files  and  3. 
Subject  Matter  Expert  (SME)  opinion. 

•  All  databases  referenced  were  assumed  to  contain  valid  anthropometric  information;  No 
cost  of  insuring  data  validity  was  included  in  the  individual  estimates. 

•  The  domain  of  users  was  limited  to  inside  the  Air  Force  Civilian,  Military  and  Contractor 
personnel. 

•  Databases  used  will  be  those  the  Air  Force  currently  has  in  its  possession,  including  some 
Army,  Navy,  and  international  sources,  though  these  may  not  be  the  most  recent. 

•  No  schedule  or  budget  limitations  were  known  to  exist. 

•  Costs  associated  with  maintenance  and  updates  to  the  referenced  databases  were  assumed 
to  be  equal  for  each  alternative  and  therefore  were  not  included  in  the  individual 
estimates. 

•  Technological  solutions  peripheral  to  the  agreed  upon  alternatives  were  not  addressed. 

2.4  COST  ESTIMATE 
2,4.1  Process 

For  each  alternative,  technical,  schedule  and  programmatic  ground  rules  and  assumptions 
were  established.  Based  on  this  information  an  estimating  methodology  was  selected  and  data 
was  gathered.  A  cost  estimate  was  prepared  for  each  alternative  and  was  then  normalized  to 
constant  year  dollars  for  cost  comparison  purposes. 
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2.4.2  Evaluation 


2.4.2. 1  Alternative  #1,  Status  Quo 


In  addition  to  the  global  groundrules  and  assumptions,  the  following  assumptions  were 
made  in  preparing  the  cost  estimate  for  this  alternative: 


•  AFRL/HE  receives  an  average  of  one  question  per  week  (50  per  year). 

•  It  takes  an  average  of  240  hours  to  completely  answer  each  inquiry  (For  example, 
pictures  are  reviewed,  site  visits  are  made,  research  is  conducted,  data  is  analyzed  and 
answer  is  provided). 

•  The  average  of  a  GS-14  and  GS-15  salary  is  assumed  to  be  representative  of  the  salary 
level  of  the  variety  of  personnel  who  spend  time  answering  the  inquiries. 

•  The  cost  of  contract  management  labor  required  to  oversee  the  existing  data  collection 
contract  is  assumed  to  be  equivalent  to  15%  of  one  GS-12  annually. 

•  The  sources  utilized  for  anthropometric  information  do  not  cost  anything  to  access. 

•  There  are  no  "non-organic"  sources  of  information  required. 

•  One  half  of  the  questions  posed  to  AFRL/FIEC  require  a  site  visit  to  properly  answer 
them.  For  purposes  of  this  estimate,  a  typical  trip  is  3  days  in  length  and  a  typical 
location  is  San  Antonio  Texas. 

•  The  cost  of  annual  software  maintenance  and  upgrades  required  to  support  this  scenario 
is  $50K  annually. 

•  Technical  support  such  as  data  management,  backups,  equipment  and  facility 
maintenance  required  is  assumed  to  be  equivalent  to  1  Opercent  of  the  labor  support. 

•  Annual  hardware  requirements  are  replacement/upgrade  of  1 0  percent  of  the  current 
systems  each  year.  Specifically,  two  state-of-the-art  PC's  (one  50  gigabyte  or  larger), 
miscellaneous  lab  equipment  and  the  FARO  arm.  Estimate  includes  $50K  per  year  to 
cover  these  costs. 

•  There  was  assumed  to  be  a  cost  of  $1K  annually  for  consumables  associated  with  the 
delivery  of  technical  reports  (paper  and  CD's) 


Table  1  summarizes  the  costs  of  alternative  #1. 


Table  1.  Alternative  #1  Costs  (Constant  FY02  $000) 


Costs  * 

FY02 

FY03 

FY04 

FY05 

Total 

System  Acquisition 

0 

0 

0 

0 

0 

System  Implementation 

13.25 

13.25 

13.25 

13.25 

66.23 

Operation  and  Support 

983.70 

983.7 

983.7 

983.7 

4,918.5 

Alternative  #1  Total 

996.95 

996.95 

996.95 

996.95 

4,984,73 

*  A  cost  estimating  methodology  matrix  and  detailed  worksheet  can  be  found  in  Appendix  A. 
Numbers  may  not  add  due  to  rounding. 


2.4.2.2  Alternative  #2,  Searchable  Database 

In  addition  to  the  global  groundrules  and  assumptions,  the  following  assumptions  were 
made  in  preparing  the  cost  estimates  for  this  alternative: 

•  The  database  will  be  a  web  based  Oracle  system  application. 

•  The  searchable  database  will  contain  a  statistics  package  capability  (SAS). 

•  The  data  retrieved  from  electronic  sources  will  be  importable  into  the  statistics  package. 


•  The  total  population  of  potential  users  is  500;  the  maximum  possible  at  any  one  time  is 
50. 

•  There  are  approximately  40  separate  sources  of  information  ranging  from  ASCII  flat  files 
to  3D  databases.  SMEs  estimate  it  will  require  one  person  (GS-12  equivalent)  working 
full  time  for  one  year  to  collate  all  sources,  solve  potential  integration  issues  and  sort  out 
what  should  be  included  in  the  database  as  sources  of  information.  Activities  include 
figuring  out  the  fields  required,  cross-referencing  data  from  all  the  different  sources  and 
putting  them  in  a  consistent,  standard  format.  Additional  information  on  the  primary  data 
sources  follows: 

•  Cockpit  Accommodation:  Moderate  file  size  in  MS  Access. 

•  CAESAR:  Unix-based  cyberware  format  from  silicon-graphics  scans.  File 
size=3.5MB  compressed. 

•  CG  Whole  Body:  Unix-based  cyberware  format  from  silicon-graphics  scans.  File 
size=3.6MB  compressed. 

•  Pregnant  Woman:  Unix-based  cyberware  format  from  silicon-based  graphics  scans. 
File  size=3.5MB  compressed. 

•  3D  Head:  Unix-based  cyberware  format  from  silicon-based  graphics  scans.  File 
size=434  kB  compressed. 

•  30-1-  miscellaneous  text-based  ASCII  formatted  files  minimal  in  size. 

•  There  will  be  no  requirement  for  classified/proprietary  information. 

•  All  hardware  and  network  requirements  to  host  this  alternative  are  currently  in  place  (PC 
server  with  multiple  hard  drives  of  50  gigabytes  or  more  each  and  a  Finux  based  server 
linked  to  graphics  workstations) 

•  The  purchase  of  the  Oracle  object-oriented  database  management  software  is  required. 

•  The  current  communication  links  have  appropriate  bandwidth  to  support  operation  of  this 
application. 

•  The  overall  timeline  is  assumed  to  be  representative  of  the  project  schedule  outlined  in 
Appendix  A. 

•  Annual  hardware  requirements  are  replacement/upgrade  of  1 0  percent  of  the  current 
systems  each  year  —  specifically,  two  state-of-the-art  PCs  (one  50  gigabyte  or  larger), 
miscellaneous  lab  equipment  and  the  FARO  arm.  Estimate  includes  $50K  per  year  to 
cover  these  costs. 

•  The  cost  of  annual  software  maintenance  and  upgrades  required  to  support  this  scenario 
is  $50K  annually. 

•  Project  management  is  required  during  system  implementation.  1  GS- 14  equivalent  one- 
half  time  during  the  first  year. 

•  The  effort  required  to  develop  a  training  curriculum  requires  two  people  (average  grade 
GS-14)  one  half  time  for  a  three-year  period.  In  addition,  1  person  (GS-12)  is  required 
one-half  time  during  the  last  year  to  finalize  all  course  materials 

•  User  training  will  be  offered  at  Wright-Patterson  Air  Force  Base  (WPAFB),  Ohio,  three 
times  per  year  to  an  average  of  20  students  per  class.  The  length  of  the  course  will  be 
equivalent  to  a  3 -credit  masters  level  course  (45  hours  of  instruction)  and  will  be  taught 
by  a  GS-14. 

•  Course  materials  for  the  training  are  estimated  to  cost  $25  per  student  (textbook  and  CD). 

•  Travel  and  per  diem  costs  to  attend  the  training  will  be  covered  by  the  student  and  are 
therefore  not  included  in  this  estimate. 

•  There  will  continue  to  be  a  requirement  for  the  status  quo  alternative  while  users  become 
trained  on  this  searchable  database.  SME’s  estimate  the  requirement  for  individual  calls 
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to  AFRL/HE  will  decrease  as  follows:  1®*  year:  -25  percent,  2°^  year:  -50  percent,  3'“* 
year:  -75  percent. 

•  Annual  support  requirements  (data  systems  manager,  troubleshooter,  maintainer  of 
system  security,  provider  of  backups  and  updates  to  the  system,  etc)  for  this  alternative 
are  for  one  full  time  dedicated  support  technician  (GS-12). 

Table  2  summarizes  the  costs  for  alternative  #2. 


Table  2.  Alternative  #2  Costs  (Constant  FY02  $000) 


Costs  * 

FY02 

FY03 

FY04 

FY05 

FY06 

Total 

System  Acquisition 

323.6 

1,625.1 

214.7 

0 

0 

2,163.36 

System  Implementation 

60.4 

370.5 

171.3 

0 

0 

602.2 

Operation  &  Support 

983.7 

983.7 

1,228.3 

1,040.9 

734.3 

4,971.0 

Alternative  #2  Total 

1,367.7 

2,979.3 

1,641.3 

1,040.9 

734.3 

7,736.6 

*  A  cost  estimating  methodology  matrix,  detailed  cost  worksheet  and  project  schedule  can  be 
found  in  Appendix  A.  Numbers  may  not  add  due  to  rounding. 

2.4.2.3  Alternative  #3,  Searchable  Database  with  Expert  Interface 

In  addition  to  the  global  groundrules  and  assumptions,  the  following  assumptions  were 
made  in  preparing  the  cost  estimate  for  this  alternative: 

•  All  assumptions  from  alternative  #2  apply  here  except  no  training  is  required  for  the  user. 

•  The  overall  timeline  is  assumed  to  be  representative  of  the  project  schedule  outlined  in 
Appendix  A. 

•  Five  basic  “templates”  will  be  offered  in  the  “expert”  database  function: 

•  Generation  of  a  requirements/specification  document. 

•  Development  of  a  prototype. 

•  Evaluation  of  an  existing  system. 

•  Queries  regarding  who  fits  in  what  systems. 

•  Flow  to  enter  new  data  into  the  database. 

•  Each  template  will  have  the  following  basic  steps: 

•  Statement  of  the  problem. 

•  Identification  of  the  environment  and  human  constraints. 

•  Selection  of  relevant  dimensions. 

•  Definition  of  a  target  population. 

•  Determination  of  appropriate  sample  of  the  population. 

•  Selection  of  cases  from  the  sample  to  characterize  the  population. 

•  Implementation  of  the  cases  into  the  output  format. 

•  The  average  number  of  data  sources  required  to  obtain  a  solution  for  each  template  is 
five. 


Table  3  summarizes  the  costs  for  alternative  #3. 

Table  3.  Alternative  #3  Costs  (Constant  FY02  $  in  000) 


Costs  * 

FY02 

FY03 

FY04 

FY05 

FY06 

Total 

System  Acquisition 

721.1 

4,123.8 

1,227.1 

0 

0 

6,072.0 

System  Implementation 

45.7 

134.5 

173.8 

0 

0 

354.0 
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Operation  and  Support 

983.7 

983.7 

1,959.3 

1,902.4 

1,195.6 

7,024.7 

Alternative  #3  Total 

1,750.5 

5,242.0 

3,360.3 

1,902.4 

1,195.6 

13,450.7 

*  A  cost  estimating  methodology  matrix,  detailed  cost  worksheet  and  project  schedule  can  be 
found  in  Appendix  A.  Numbers  may  not  add  due  to  rounding. 


2.5  SENSITIVITY  ANALYSIS 

2.5.1  Process 

The  next  step  of  a  cost  benefit  analysis,  the  sensitivity  analysis,  illustrates  how  changes  in 
key  assumptions  and  variables  within  the  cost  estimate  may  have  an  impact  on  the  total  cost 
estimate.  These  changes  are  often  referred  to  as  "excursions."  They  answer  the  question,  "What 
elements,  if  changed,  would  influence  the  overall  cost  of  each  alternative?"  The  result  of  such  an 
excursion  assesses  the  magnitude  of  change  required  within  key  cost  elements  sufficient  to 
influence  the  outcome  of  the  cost  analysis.  While  many  different  excursions  could  be  evaluated, 
the  ones  most  likely  to  be  of  interest  to  the  reviewers  of  the  analysis  are  selected. 

2.5.2  Evaluation 

The  sensitivity  analysis  focuses  on  two  areas.  First  is  the  dependence  on  expert  opinion 
to  estimate  the  software  development  effort  common  to  both  alternatives  #2  and  #3  and  second  is 
the  method  used  to  estimate  the  expert  interface  function  unique  to  alternative  #3. 

While  alternatives  #2  and  #3  differ  significantly  in  terms  of  total  cost,  both  contain  the 
same  cost  element.  Software  Investment.  The  software  investment  element  is  composed  of 
software  licenses  and  a  software  development  effort.  Specifically,  the  software  development 
effort  makes  up  a  significant  portion  of  the  cost  for  both  alternatives:  24  percent  of  alternative  #2 
and  44  percent  of  alternative  #3. 

The  portion  of  the  software  investment  element  that  is  common  to  alternatives  #2  and  #3 
is  the  development  of  an  oracle  database  and  the  incorporation  of  a  statistical  analysis  tool.  Both 
alternatives  #2  and  #3  assume  the  use  of  an  Oracle  database  with  1,000  to  2,000  function  points 
(40,000  to  80,000  lines  of  code)  and  another  100  to  200  function  points  (3,200  to  6,400  lines  of 
code)  for  the  SAS  integration.  The  SEER/SEM  software  cost  estimating  tool  was  used  to 
estimate  these  components.  The  inputs  for  the  model  were  gathered  from  anthropometric  experts 
and  considered  the  respective  sizes  of  the  existing  files  to  be  integrated  into  the  ACCES  system. 
Since  this  portion  of  the  estimate  is  the  same  for  both  alternatives  ($1.9M),  variations  in  the  input 
variables  may  change  the  total  cost,  but  would  not  impact  the  comparative  analysis  between  these 
two  alternatives. 

The  key  difference  between  alternatives  #2  and  #3  is  the  inclusion  of  an  expert  interface 
function  for  the  third  alternative  (+$3.8M).  The  estimate  for  this  portion  of  alternative  #3  was 
based  largely  on  information  obtained  from  anthropometric  experts  envisioning  how  an  expert 
system  could  deliver  accurate  anthropometric  information.  The  total  cost  is  sensitive  to  the  inputs 
received  from  these  experts  who  attempted  to  capture  this  function  in  terms  of  templates  that  will 
simulate  possible  anthropometric  scenarios.  Flowever,  even  if  this  particular  cost  element  was 
overstated  by  as  much  as  50  percent,  the  total  cost  of  alternative  #3  would  still  exceed  the  cost  of 
alternative  #2.  The  effect  of  this  area  of  sensitivity  will  be  further  discussed  in  the  risk  analysis 
section  (section  2.8)  and  combined  with  the  impact  of  the  benefits  analysis  to  determine  a 
potential  influence  on  the  overall  ranking  of  alternatives. 

The  primary  source  of  information  for  all  three  cost  estimates  was  subject  matter  experts 
within  the  Air  Force  anthropometric  community.  To  the  extent  possible,  costs  were  estimated 
based  on  known  variables  (e.g.,  AFRE/FIE  labor  costs,  current  network  architecture,  hardware 
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and  software  budgets,  etc.).  However,  since  alternative  #rs  cost  estimate  was  based  completely 
on  these  documented  variables,  it  was  not  addressed  in  this  sensitivity  analysis. 

2.6  BENEFITS  CHARACTERIZATION  AND  VALIDATION 


2.6.1  Process 


Analysis  of  the  benefits  was  accomplished  using  a  decision  support  tool.  Expert  Choice, 
which  enabled  Human  Systems  lAC  to  leverage  the  expertise  of  key  stakeholders  and  experts  in 
the  field.  As  stated  in  Section  2.3,  the  specific  goal  of  the  alternatives  examined  was  to  develop 
an  anthropometric  size,  shape,  fit  and  accommodation  information  system  for  efficient  and 
effective  use  by  designers,  manufacturers,  buyers,  and  decision  makers  across  a  number  of 
domains.  Key  results  of  the  alternatives  under  scrutiny  were  an  appropriate  and  accurate 
anthropometric  information  system  that  allowed  designers  to  provide  for  the  selection  of 
appropriate  users.  Additional  results  included  understandable,  adaptable  anthropometric  and 
accommodation  information  for  the  user. 

Benefits  were  identified  using  a  combination  of  bottom-up  and  top-down  analyses.  First, 
Human  Systems  lAC  identified  any  and  all  terms  or  phrases  that  could  be  associated  with  an 
improvement  or  decrement  in  any  of  the  three  identified  investment  alternatives.  In  the  case  of 
ACCES,  a  list  of  over  40  possible  benefits  was  initially  developed.  This  bottom-up  approach  was 
intended  to  tease  out  any  and  all  benefits  of  the  three  alternative  systems.  The  resulting  list  was 
then  segregated  into  categories  of  benefits.  For  instance,  any  benefit  that  influenced  the  ease  of 
use  of  the  system  was  included  in  the  "usability"  category. 

The  second  step  was  to  employ  a  top-down  analysis  on  the  benefits  list.  Using  the 
overall  goal  (see  definitions  section)  as  a  starting  point.  Human  Systems  lAC  categorized  the 
global  areas  of  improvement  or  decrement  that  would  realize  a  difference  between  the 
alternatives.  For  instance,  the  scope  of  information  available  was  a  major  difference  between 
alternatives  and  was  identified  as  a  possible  benefit. 

The  final  step  in  identifying  alternatives  was  to  compare  the  bottom-up  with  the  top- 
down  list.  The  top-down  list  described  the  global  benefits  and  differences  between  investment 
alternatives.  The  bottom-up  list  described  the  specifics  that  should  be  represented  by  the  global 
benefits.  Any  bottom-up  benefits  that  were  not  represented  in  the  global  list  were  considered  as  a 
unique  benefit.  Any  top-down  benefit  that  was  not  well  represented  by  specific  instances  from 
the  bottom-up  list  was  reevaluated  as  a  benefit.  In  the  end,  five  global  benefits  were  identified 
using  this  combination  approach.  They  were  accessibility,  accuracy,  efficiency,  scope,  and 
usability. 

The  Expert  Choice  session  began  by  reaching  a  general  consensus  on  the  definitions  for 
the  goal  of  the  session,  the  three  investment  alternatives,  and  the  five  associated  benefits  to  be 
evaluated  during  the  session.  Definitions  played  a  critical  role  since  evaluations  made  by  the 
participants  were  based  on  their  interpretations  of  those  definitions.  The  Expert  Choice  process 
used  in  this  approach  is  illustrated  in  Figure  1 . 


k  Identify  \ 

K  Goal,  \  Rank 

A  Benefits  )  All 

W  and  /  Benefits 

W  Aiternatives  / 


Rank  \  „  ,  ,  \  Determine  ^ 

\  Aiternatives  \  Caicuiate  \  preferred 
for  each  /  lAiternative  , 

Benefit  /  j  q  ^ 


/ 


/ 


Expert  Choice  Process 
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Once  the  definitions  were  clearly  defined  and  understood  by  all  participants,  the  Expert 
Choice  group  ranked  all  five  of  the  benefits  in  order  of  importance.  For  instance,  accuracy  is 
more  important  than  accessibility,  and  accessibility  is  more  important  than  efficiency.  Based  on 
the  group's  responses,  the  Expert  Choice  tool  calculated  the  benefit  scores  and  attached  a  value  to 
the  benefits  and  alternatives.  The  result  was  that  the  highest  score  determined  the  alternative  with 
the  most  overall  benefit  value. 

Then,  the  group  ranked  the  three  investment  alternatives  from  highest  to  lowest 
importance  for  each  benefit  individually.  As  a  result,  the  three  alternatives  were  compared  five 
times,  once  for  each  benefit.  The  alternative  ranked  with  the  highest  importance  meant  that  it 
realized  the  highest  value  for  that  benefit.  For  instance,  if  alternative  #2  had  the  best  usability 
features,  according  to  the  Expert  Choice  group,  then  it  would  have  been  rated  highest  on 
usability. 

2.6.2  Evaluation 

Stakeholders  identified  by  Human  Systems  lAC  and  the  customer  participated  in  the 
evaluation.  Among  these  stakeholders  were  government  and  civilian  SMEs  in  anthropometry 
from  AFRE,  ASC,  ACC,  Human  Systems  SPO,  and  Boeing.  Table  4  lists  the  participants  and 
their  affiliation. 


Table  4.  Expert  Choice  Participants 


Participant 

Affiliation 

James  M.  Bamaba 

ASC/ENFC 

Eric  Crawford 

ASC/ENFC 

Alfonso  Gonzalez 

Human  Systems  SPO,  311  HSW/YA 

CMSgt  Randolph  C.  Eoving 

ACC/DRSR 

Teresa  Perkins 

ASC/ENFC 

Kathleen  M.  Robinette 

AFRE/HECP 

Edward  R  Winkler 

Boeing 

Gregory  F.  Zehner 

AFRE/HECP 

As  stated  in  previous  sections,  three  alternatives  were  evaluated: 

•  Alternative  #1  Status  Quo, 

•  Alternative  #2  Searchable  Database,  and 

•  Alternative  #3  Searchable  Database  with  Expert  Interface 

Each  of  the  three  identified  investment  alternatives  was  evaluated  across  five  benefits  using 
Expert  Choice.  Table  5  lists  the  alternatives  along  with  the  abbreviation  used  in  the  Expert 
Choice  analysis. 


Table  5.  Alternative  Abbreviations 


Expert  Choice 
Abbreviation 

Alternative  Name 

STAT  QUO 

Status  quo;  inquiries  to  AFRE/HE 

DAT'BASE 

Searchable  database 

DBW/EXP 

Searchable  database  with  expert  interface 
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The  five  global  benefits  were  identified  through  a  thorough  review  of  the  requirements, 
the  literature,  and  SME  input.  Table  6  lists  the  benefits,  their  abbreviations  used  in  the  Expert 
Choice  model,  and  a  brief  definition  used  in  this  analysis. 
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Table  6.  Benefits  Abbreviations  and  Definitions 


Expert 

Choice 

Abbreviation 

Benefit  Name 

Benefit  Definition 

ACCESS 

Accessibility 

System  impact  on  ease  of  accessing  anthropometric 
information;  the  number  and  type  of  individuals  that  can  locate 
the  info. 

ACCURACY 

Accuracy 

System  impact  on  currency,  correctness,  and  direct  use  of 
fit/accommodation  information.  Information  provided  is  the 
correct  solution  for  the  task  at  hand. 

EFFICIEN 

Efficiency 

System  impact  on  speed  of  finding  anthropometric 
fit/accommodation  information.  A  more  efficient  system  will 
require  less  time  invested. 

SCOPE 

Scope 

System  impact  on  sufficiency  of  anthropometric 
fit/accommodation  information  provided  to  user.  Information 
provided  is  in-depth  enough  to  meet  user's  needs. 

USABE'TY 

Usability 

Ease  of  system  use  in  accessing  anthropometric 
fit/accommodation  information.  Eevel  of  effort  required  of 
designer  to  find  information.  A  highly  useable  system  requires 
little  effort. 

Benefits  were  evaluated  using  pairwise  comparisons  to  "prioritize"  the  importance  of 
each  with  respect  to  the  goal.  Table  7  shows  the  results  of  each  comparison. 


Table  7.  Global  Table  of  Pairwise  Comparisons 


ACCESS 

9 

8 

7 

6 

5 

4 

3 

2 

1 

2 

3 

4 

5 

6 

@ 

8 

9 

ACCURACY 

ACCESS 

9 

8 

7 

6 

4 

3 

2 

1 

2 

3 

4 

5 

6 

7 

8 

9 

EFFICIEN 

ACCESS 

9 

8 

7 

6 

5 

4 

3 

2 

1 

2 

3 

4 

5 

® 

7 

8 

9 

SCOPE 

ACCESS 

9 

8 

7 

6 

5 

4 

3 

2 

1 

(D 

3 

4 

5 

6 

7 

8 

9 

USABE'TY 

ACCURACY 

9 

8 

(2) 

6 

5 

4 

3 

2 

1 

2 

3 

4 

5 

6 

7 

8 

9 

EFFICIEN 

ACCURACY 

9 

8 

7 

6 

(D 

4 

3 

2 

1 

2 

3 

4 

5 

6 

7 

8 

9 

SCOPE 

ACCURACY 

9 

8 

@ 

6 

5 

4 

3 

2 

1 

2 

3 

4 

5 

6 

7 

8 

9 

USABE'TY 

EFFICIEN 

9 

8 

7 

6 

5 

4 

3 

2 

1 

2 

3 

4 

5 

7 

8 

9 

SCOPE 

EFFICIEN 

9 

8 

7 

6 

5 

4 

3 

2 

1 

© 

3 

4 

5 

6 

7 

8 

9 

USABE'TY 

SCOPE 

9 

8 

(7) 

6 

5 

4 

3 

2 

1 

2 

3 

4 

5 

6 

7 

8 

9 

USABE'TY 

Ratings:  1  =  Equal;  3  =  Moderate;  5  =  Strong;  7  =  Very  Strong;  9  =  Extreme 

Example  of  interpretation:  Accuracy  (ACCURACY)  is  strongly  more  important  than  scope 

(SCOPE). 


While  Table  7  shows  the  global  comparisons,  Table  8  adds  the  rank  order,  individual 
score,  and  summarizes  these  comparisons  into  a  single  table.  As  can  be  seen  in  Table  8,  the 
Expert  Choice  group  determined  that  accuracy  was  the  most  important  benefit,  and  scope  was  the 
second  most  important.  Accessibility  and  usability  followed  as  third  and  fourth  and  had  about  the 
same  value.  Efficiency  was  ranked  the  least  important  of  the  benefits. 
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Table  8.  Rank  Order  of  Benefits 


Rank 

Benefit 

Benefit  Rating 
(Score  Out  of  1.0) 

1 

Accuracy 

0.552 

2 

Scope 

0.268 

3 

Accessibility 

0.073 

4 

Usability 

0.071 

5 

Efficiency 

0.037 

2. 6.2.1  Evaluation  of  Alternatives  for  Benefits 

Supported  by  the  information  in  Appendix  B,  the  Expert  Choice  team  discussed  the 
positive  and  negative  aspects  of  each  alternative.  Their  inputs  were  distilled  into  a  single  ranking 
for  each  alternative  across  each  benefit.  These  rankings  were  then  used  by  Expert  Choice  to 
complete  the  benefits  analysis. 

Based  on  the  results  from  the  benefits  analysis,  shown  in  Table  9,  the  most  effective 
solution  was  the  searchable  database  with  expert  interface  (63.3  out  of  100),  with  the  searchable 
database  falling  into  the  middle  (29.7).  The  least  effective  alternative  with  respect  to  benefits  was 
the  Status  Quo  system,  which  scored  only  7.0  out  of  100. 


Table  9.  Alternative  Benefit  Scores 


Alternative 

Benefit  Score 

Status  Quo  System 

7.0 

Searchable  Database 

29.7 

Searchable  Database  with  Expert  Interface 

63.3 

2,7  NET  VALUE  ANALYSIS 

2.7.1  Process 

The  net  value  analysis  is  the  combination  of  life  cycle  costs  developed  in  the  cost 
assessment  with  the  alternative  benefits  established  in  the  benefits  assessment.  Included  in  this 
portion  of  the  report  is  the  final  cost  benefit  ratio  that  will  act  as  the  foundation  of  the 
recommendations  provided.  The  cost  benefit  ratio  is  simply  a  combination  of  the  costs 
(numerator)  and  the  benefits  (denominator)  into  a  ratio.  For  example,  the  CBR  for  the  first 
alternative  would  be: 

CBR  Alternative  1  =  Cost  of  Alternative  1 

Benefit  Score  of  Alternative  1 


2.7.2  Evaluation 

As  stated  in  Section  2.6,  the  Benefits  Evaluation,  the  overall  benefit  ratings  by  subject 
matter  experts  and  stakeholders  rated  alternative  #1,  the  status  quo,  with  a  score  of  7.0. 
Alternative  #2,  the  searchable  database  scored  29.7,  and  alternative  #3,  the  searchable  database 
with  expert  interface  scored  63.3.  When  combined  with  the  costs,  we  see  that  alternative  #3,  with 
a  CBR  of  210.5,  has  the  greatest  value  when  costs  and  benefits  were  combined.  Alternative  #2 
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has  slightly  less  value,  with  a  CBR  of  256.3,  and  alternative  #1  has  the  least  value  with  a  CBR  of 
712.1. 


Table  10.  Ranking  of  Alternatives  Using  Cost  Benefit  Ratios 


Alternative 

Cost 

(Constant  $000) 

Benefit 

Score 

Cost/Benefit 

Ratio 

Ranking 

#1  Status  Quo 

$4,918.5 

7.0 

702.6 

3 

#2  Searchable  D-Base 

$7,736.6 

29.7 

260.5 

2 

#3  Searchable  D-Base  w/  Expert  Int. 

$13,450.7 

63.3 

212.5 

1 

Based  on  this  data,  alternative  #3  is  the  most  cost-effective.  The  analysis  indicates  it 
provides  the  greatest  benefit  for  the  money,  with  respect  to  the  stated  goal.  Alternative  #1  ranked 
the  least  cost-effective  and  alternative  #2  fell  in  the  middle. 

2.8  RISK  ANALYSIS 

2.8.1  Process 

In  the  final  step  of  a  cost  benefit  analysis,  both  costs  and  benefits  are  investigated  from  a 
risk  perspective.  The  objective  of  this  step  is  to  isolate  areas  of  the  alternatives  where 
uncertainties  exist  in  the  analyses.  Program  managers  then  have  the  added  foresight  to  focus  their 
attention  on  those  risky  areas  when  making  programmatic  decisions.  In  general,  the  more 
complex  the  alternative,  the  more  likely  it  is  that  changes  in  schedule/cost,  or  benefit  assumptions 
will  result.  For  example,  if  certain  cost  assumptions  or  benefit  assessments  change  dramatically, 
the  overall  findings  will  likely  change  as  well.  These  possible  areas  of  risk  are  highlighted  in  the 
evaluation  below. 

2.8.2  Evaluation 

The  areas  of  risk  for  costs  and  benefits  will  be  addressed  separately.  From  a  cost 
perspective,  there  are  two  areas  that  are  addressed  for  their  potential  risk  to  the  overall  analysis. 
These  are  the  application  of  expert  interface  technology  in  the  area  of  anthropometry  and  the 
impact  of  expert  opinion  as  a  primary  data  source. 

As  discussed  in  the  sensitivity  analysis,  a  significant  portion  of  the  total  cost  of 
alternatives  #2  and  #3  relate  to  the  cost  element.  Software  Investment.  Specifically,  the 
introduction  of  an  expert  interface  function  into  alternative  #3  results  in  an  increase  of  $3.8M 
over  alternative  #2  for  this  element  alone.  While  the  incorporation  of  the  expert  interface 
function  was  estimated  using  inputs  from  extremely  knowledgeable  experts  in  the  field  of 
anthropometry,  the  actual  application  of  an  expert  interface  in  this  milieu  is  unproven.  Should  the 
input  variables  used  to  estimate  this  portion  of  alternative  #3  vary  significantly,  the  ranking  of 
alternatives  may  change.  To  summarize  the  risk,  even  though  the  total  cost  of  alternatives  #2  and 
#3  vary  significantly,  alternative  #3's  cost  would  only  have  to  increase  by  22  percent  in  order  for 
the  cost  benefit  ratio  to  change  enough  to  impact  the  overall  ranking  of  alternatives.  That  is,  if 
the  cost  of  alternative  #3  increased  by  22  percent,  it  would  change  the  cost  benefit  ratio  enough  o 
impact  the  ranking  of  alternatives.  Thus,  should  the  cost  increase  be  realized,  alternative  #3 
would  no  longer  be  the  most  desirable  solution.  This  makes  the  accuracy  of  the  expert  interface 
estimate  crucial  to  the  credibility  of  the  overall  rankings.  As  a  result,  this  area  may  require 
additional  research  addressing  the  specific  impact  on  the  cost  of  the  alternatives  at  that  time. 

The  primary  source  of  cost  data  for  the  ACCES  alternatives  was  expert  opinion.  Some  of 
the  elements  estimated  using  this  source  were  based  on  known  values  (e.g.,  labor  costs,  hardware 
and  software  budgets,  etc.),  while  others  required  the  experts  to  envision  a  computerized 
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capability  that  is,  as  of  yet,  unproven.  In  defense  of  this  source's  accuracy,  the  data  gathering 
efforts  used  in  developing  the  cost  estimate  combined  the  talents  of  experienced  Information 
Technology  experts  and  personnel  with  extensive  knowledge  of  anthropometric  data  and  the 
process  required  to  utilize  it  appropriately.  We  are,  however,  confident  that  the  effort  resulted  in 
estimates  that  are  the  most  accurate  reflection  possible  of  the  potential  cost  of  implementing  each 
of  these  solutions. 

In  addition  to  the  two  areas  of  cost  risk,  two  areas  of  risk  associated  with  the  benefits 
assessment  were  identified.  The  first  area  of  risk  was  the  absence  of  practical  experience  by  the 
experts  assessing  with  one  or  more  of  the  alternatives  being  assessed.  The  second  area  relates  to 
the  fact  that  no  novice  users  were  included  in  the  pool  of  experts  conducting  the  analysis  of 
benefits. 

Given  that  alternative  #1  is  the  status  quo,  and  the  underlying  technology  supporting 
alternative  #2  has  been  successfully  applied  in  the  real  world,  there  was  little  risk  associated  with 
the  evaluation  of  their  benefits.  The  risk  emerged  in  the  third  alternative,  searchable  database 
with  an  expert  interface.  The  potential  variability  was  due  to  the  fact  that  assessment  of  benefits 
related  to  this  solution  had  to  rely  on  the  defined  alternative  and  its  expected  successful 
application  in  the  anthropometric  community.  While  searchable  databases  are  used  often,  a  true 
"expert  interface"  is  a  new  and  uncertain  technology.  As  a  result,  there  is  risk  that  the  technology 
may  not  work  as  expected  when  fully  developed  and  used  in  this  environment. 

The  group  that  participated  in  the  benefits  assessment  had  extensive  experience  in  the 
areas  of  interest  and  was  made  up  of  both  government  and  industry  experts.  However,  as  with  the 
assessment  of  many  benefits,  the  subjective  opinions  of  participants  formed  the  foundation  of  the 
benefits  analysis.  If  a  new  Expert  Choice  group  was  assembled,  it  is  possible  that  the  alternatives 
may  be  ranked  in  a  different  way,  based  on  their  own  personal  experiences  and  opinions. 
Specifically,  while  an  expert's  focus  may  be  on  accuracy  and  scope  of  information,  they  may  not 
take  into  consideration  the  needs  of  the  common  user.  Due  to  this  fact,  the  possibility  exists  that 
the  rating  of  some  benefits  may  be  slightly  skewed  toward  the  expert  user  and  not  the  novice.  For 
instance,  "usability"  and  "accessibility"  may  have  held  greater  sway  for  a  novice  user  who  is 
simply  trying  to  locate  information.  While  worthy  of  mention,  this  possible  change  is  unlikely  to 
shift  the  overall  ranking  of  the  three  alternatives  due  to  the  fact  that  alternative  #3,  the  searchable 
database  with  expert  interface,  would  be  the  most  user-friendly  of  all  the  alternatives.  This 
observation  in  no  way  negates  the  validity  of  the  conducted  assessment,  but  should  be  noted  as  a 
factor  to  consider  when  making  programmatic  decisions  based  on  the  conclusions  contained  in 
this  report. 
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3.  CONCLUSIONS  AND  RECOMMENDATIONS 


This  study  evaluated  the  costs  and  benefits  associated  with  three  anthropometric 
information  alternatives.  Each  alternative  was  estimated  for  cost  and  evaluated  for  relative 
benefits.  By  assigning  a  numeric  value  to  the  benefits,  Human  Systems  lAC  was  able  to  develop 
a  cost/benefit  ratio  (CBR)  for  each  of  the  three  alternatives  and  document  the  resulting  rank  order 
based  on  this  CBR. 

The  most  effective  alternative  with  respect  to  the  stated  goals  was  alternative  #3,  the 
searchable  database  with  expert  interface  (CBR  =  212.5).  The  second  most  desirable  choice  was 
alternative  #2,  the  searchable  database  (CBR  =  260.5).  Based  on  the  evaluated  cost  and  benefits 
the  least  cost-effective  solution  for  providing  anthropometric  information  was  alternative  #1,  the 
status  quo  (CBR  =  712.1). 

In  conclusion,  the  use  of  inaccurate  or  incomplete  anthropometric  information  can  lead  to 
decreased  safety,  inefficient  procurement  of  materiel,  and  ill-fitting,  difficult  to  use  man-mounted 
equipment.  Therefore,  there  must  be  a  way  for  designers  of  crew  stations  and  equipment  to 
design  for  the  user  and  provide  for  the  selection  of  appropriate  users,  based  on  effective 
application  of  anthropometric  data.  Based  on  the  results  of  this  cost  benefit  analysis,  it  is  clear 
the  effort  to  develop  ACCES  will  help  decision  makers  make  educated  judgments  about 
accommodation  information  systems  across  a  number  of  domains. 
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APPENDIX  A,  COST  ELEMENT  STRUCTURE,  MATRICES  &  ESTIMATE  DETAIL 


Air  Combat  Capability  Enhancement  Suite  (ACCES) 
Cost  Element  Structure  for  all  Alternatives 


Element 

Description 

1.0  System  Acquisition 

Roll-up  element 

1 . 1  Aequisition  Support 

Roll-up  element 

1.1.1  Proj  eet  Management 

Technical  and  business  management  effort  expended  in  the  process  of 
planning,  analyzing,  designing,  and  acquiring  the  system 

1.1.2  RFP  Preparation 

Effort  expended  in  the  process  of  developing,  refining,  and  releasing  the 
request  for  proposals  in  a  competitive  bid 

1.1.3  RFP  Evaluation  and 

Vendor  Selection 

Effort  expended  in  the  process  of  evaluating  and  comparing  RFPs  and 
selecting  the  winning  vendor 

1.2  Flardware  Investment 

Workstations,  servers,  network  devices,  peripherals,  or  other  equipment 
procured  to  support  implementation  or  operation  of  the  system 

1.3  Software  Investment 

Roll-up  element 

1.3.1  Software  Development 

Cost  of  developing,  configuring  and/or  integrating  software  applications 
to  provide  ACCES  functionality.  Includes  requirements  analysis, 
design,  coding,  testing,  and  integration 

1.3.2  Software  Licenses 

Initial,  one-time  license  costs  of  other  COTS/GOTS  software  procured 
to  support  implementation  or  operation  of  the  system 

1.4  Infrastructure  Enhancements 

Communications  upgrades  required  to  optimize  system  performance 

2.0  System  Implementation 

Roll-up  element 

2.1  Project  Management 

Technical  and  business  management  effort  expended  in  supporting, 
supervising,  and  directing  implementation  tasks  and  activities 

2.2  Contract  Management 

Effort  expended  by  contracting  staff  tracking  financials,  handling 
vendor  relations,  and  other  contract  management  responsibilities 

2.3  Hardware  Installation  &  Test 

Installation  and  testing  of  hardware  procured  for  system 

2.4  Software  Install  &  Test 

Installation  and  testing  of  software  procured  for  system 

2.5  Data  conversion 

Collation  and  integration  of  data  to  be  included  in  system.  Includes 
actual  data  entry  for  populating  database  tables 

2.6  User  Acceptance 

Effort  expended  in  formal  user  testing  of  the  production  system 

2.7  Training 

Roll-up  element 

2.7.1  Training  Development 

Costs  incurred  in  the  development  of  appropriate  training  programs  and 
material 

2.7.2  Train-the-Trainer 

Costs  associated  with  in-depth,  targeted  instruction  of  primary  training 
staff 

2.7.3  End  User  Training 

Costs  associated  with  general  instruction  of  system  end  users 
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3.0  Operations  and  Snpport 

Roll-up  element 

3.1  System  operations 

Ongoing,  dedicated  staff  support  for  operating  new  system  hardware 
and  software  (over  and  above  status  quo  operations  costs) 

3.2  Technieal  Support 

Data  management,  backups,  and  facility  maintenance 

3.3  Software  maintenance  and 
upgrades 

Recurring  software  licensing,  upgrade,  and  maintenance  costs 

3.4  Hardware  maintenance  and 
upgrades 

Recurring  hardware  upgrade,  repair,  and  maintenance  costs 

3.5  Recurring  training 

Costs  of  periodic  end  user  training,  retraining,  and  refreshment 

3.6  Other  costs 

Other  miscellaneous  costs  of  operations,  such  as  office  supplies  and 
travel  required  to  support  the  program 
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Air  Combat  Capability  Enhancement  Suite  (ACCES) 
Cost  Estimating  Methodology  Matrix 
Alternative  1 :  Status  Quo 


Element 

Estimating  Methodology 

1.0  System  Acquisition 

Assumed  no  acquisition  efforts  are  required  for  the  status  quo. 

1 . 1  Acquisition  Support 

1.2  Hardware  Investment 

1.3  Software  Investment 

1.4  Infrastructure  Enhancements 

2.0  System  Implementation 

Roll-up  element 

2.1  Project  Management 

2.2  Contract  Management 

15%  of  1  GS-12 's  labor  is  required. 

2.3  Hardware  Installation  &  Test 

2.4  Software  Install  &  Test 

2.5  Data  conversion 

2.6  User  Acceptance 

2.7  Training 

3.0  Operation  and  Support 

Roll-up  element 

3.1  System  operations 

Average  salary  ofGS-14/15;  50  questions/yr;  240  hrs/question 

3.2  Technical  support 

10%  of  System  operations 

3.3  Software  maintenance  and 
upgrades 

$50,000/yr 

3.4  Hardware  maintenance  and 
upgrades 

$50,000/yr 

3.5  Recurring  training 

Assumed  no  recurring  training  costs  are  required. 

3.6  Other  costs 

25  trips  per  year  @  $  1,029/trip;  $l,000/yr  for  consumables 
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Alternative  #1,  Status  Quo  (Constant  FY02  $  000) 


Constant  Years  (000) 

2002 

2003 

2004 

2005 

2006 

Totals 

1.0  System  Acquisition 

— 

... 

... 

... 

— 

1.1  Acquisition  Support 

... 

... 

... 

... 

... 

1.1.1  Project  Management 

... 

1.1.2  RFP  Preparation 

... 

1 .1 .3  RFP  Evaluation  and  Vendor  Selection 

... 

... 

1 .2  Hardware  Investment 

___ 

___ 

1.3  Software  Investment 

___ 

... 

... 

___ 

___ 

1.3.1  Soltware  Development 

... 

1.3.2  Sollware  Licenses 

_ 

... 

1.4  Infrastructure  Enhancements 

... 

___ 

2.0  System  Implementation 

$  13.25 

$  13.25 

$  13.25 

$  13.25 

$  13.25 

$  66.23 

2.1  Project  Management 

— 

... 

— 

__ 

— 

2.2  Contract  Management 

$  13,25 

$  13,25 

$  13,25 

$  13.25 

$  13,25 

$  66,23 

2.3  Hardware  Installation  and  Test 

___ 

___ 

— 

___ 

2.4  Software  Installation  and  Test 

___ 

___ 

— 

2.5  Data  Conversion 

___ 

... 

___ 

2.6  User  Acceptance 

___ 

--ij 

u 

___ 

2.7  Training 

___ 

___ 

___ 

___ 

2.7.1  Training  Development 

_ 

2.7.2  Train-lhe-Trainer 

2.7.3  End  User  Training 

... 

... 

3.0  Operations  and  Support 

$  983.70 

$  983.70 

$  983.701 

1  $  983.70 

$  983.70 

$  4,918.50 

3.1  Systems  Operations 

$  779.07 

$  779,07 

$  779.07 

$  779.07 

$  779,07 

$  3,895,34 

3.2  Help  Desk 

$  77,91 

$  77,91 

$  77,91 

$  77.91 

$  77,91 

$  389,53 

3.3  Software  Maintenance  and  Upgrades 

$  50,00 

$  50,00 

$  50.00 

$  50.00 

$  50,00 

$  250,00 

3.4.  Hardware  Maintenance  and  Upgrades 

$  50,00 

$  50,00 

$  50,00 

$  50.00 

$  50,00 

$  250,00 

3.5  Recurring  Training 

— 

— 

— 

— 

3.6,  Other  Costs 

$  26,73 

$  26,73 

$  26,73 

$  26.73 

$  26,73 

$  133,63 

Totals 

$  996.95T$  996.95 

$  996.95 

$  996.95 

$  996.95 

$4,984.73 
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Alternative  #1,  Status  Quo  (Then  Year  $  000) 


Then  Years  fOOO) 

2002 

2003 

2004 

2005 

2006 

Totals 

1.0  System  Acquisition 

— 

... 

... 

... 

1.1  Acquisition  Support 

... 

... 

... 

... 

... 

1 .1 .1  Protect  Manaqernent 

... 

1.1 .2  RFP  Preparation 

... 

1 .1 .3  RFP  Evaluation  and  Vendor  Selection 

... 

1.2  Hardware  Investment 

___ 

... 

... 

... 

___ 

1.3  Software  Investment 

... 

... 

... 

... 

1.3.1  Software  Development 

... 

1 .32  Software  Licenses 

_ 

... 

1.4  Infrastructure  Enhancements 

... 

... 

... 

... 

___ 

2.0  System  Implementation 

$  13.25 

$  13.64 

$  14.05 

$  14.47 

$  14.91 

$  70.33 

2.1  Project  Management 

... 

... 

... 

2.2  Contract  Management 

$  13.25 

$  13.64 

$  14.05 

$  14.47 

$  14.91 

$  70.33 

2.3  Hardware  Installation  and  Test 

___ 

... 

— 

2.4  Software  Installation  and  Test 

___ 

... 

... 

___ 

2.5  Data  Conversion 

___ 

___ 

2.6  User  Acceptance 

___ 

___ 

... 

... 

... 

2.7  Training 

... 

... 

... 

... 

... 

2.7.1  Traininq  Development 

... 

2.7,2  Train-ihe-Trainer 

... 

2.7,3  End  User  Traininq 

3.0  Operations  and  Support 

$  983.70 

$  1,013.21 

$  1,043.61 

$  1,074.92 

$  1,107.16 

$  5,222.59 

3.1  Systems  Operations 

$  779.07 

$  902.44 

$  826.51  1$  951.31 

$  876.85 

$  4.136.18 

3.2  Help  Desk 

$  77.91 

$  90.24 

$  82.65 

$  85.13 

$  87.68' 

$  413.62 

3.3  Software  Maintenance  and  Upgrades 

$  50.00 

$  51.50 

$  53.05 

$  54.64 

$  56.28 

$  265.46 

3.4.  Hardware  Maintenance  and  Upgrades 

$  50.00 

$  51.50 

$  53.05 

$  54.64 

$  56.28 

$  265.46 

3.5  Recurring  Training 

... 

... 

... 

... 

... 

... 

3.6.  Other  Costs 

$  26.73 

$  27.53 

$  28.35 

$  29.20 

$  30.03 

<J> 

00 

■rr 

Totals 

996.95 

$  1,026.85 

$  1,057.66 

$  1,089.39 

$  1,122.07 

1  $5,292.92 
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Cost  Estimating  Methodology  Matrix 
Alternative  2:  Searchable  Database 


Element 

Estimating  Methodology 

1.0  System  Acquisition 

Roll-up  element 

1 . 1  Acquisition  Support 

Roll-up  element 

1.1.1  Proj  ect  Management 

10%  of  1  GS-15’s  labor  for  8  months 

1.1.2  RFP  Preparation 

25%  of  2  GS-12s  labor  for  6  months 

1.1.3  RFP  Evaluation  and 

Vendor  Selection 

100%  of  2  GS-12s  labor  for  2  months 

1.2  Flardware  Investment 

Assumed  all  hardware  requirements  are  in  place 

1.3  Software  Investment 

Roll-up  element 

1.3.1  Software  Development 

Parametric  estimate  using  SEER-SEM  software  cost  estimating  tool. 
Average  loaded  labor  rate  $14,300  per  month. 

Assumed  development  of  an  Oracle  database  sized  between  1,000  and 
2,000  function  points  (40,000  to  80,000  source  lines  of  code) 

Platform:  Internet  Development 

Application  Type:  Object  Oriented  Database 

Acquisition  Method:  New  Development 

Development  Method:  Spiral 

Development  Standard:  Commercial 

Also  assumed  SAS  integration  of  between  100  and  200  function  points 
(3,200  to  6,400  source  lines  of  code). 

Platform:  Internet  Development 

Application  Type:  Business  Analysis  Tool 

Acquisition  Method:  New  Development 

Development  Method:  Off-the-Shelf  Integration 

Development  Standard:  Commercial 

1.3.2  Software  Licenses 

Oracle  and  SAS  licenses  for  500  users 

1.4  Infrastructure  Enhancements 

Assumed  all  network  requirements  are  in  place 

2.0  System  Implementation 

Roll-up  element 

2.1  Project  Management 

50%  of  1  GS-14’s  labor  for  18  months 

2.2  Contract  Management 

15%  of  1  GS-12’s  labor  for  20  months  (assuming  that  CM  support  will 
be  required  during  the  2-month  RFP  evaluation  period) 

2.3  Hardware  Installation  &  Test 

Pursuant  to  element  1 .2,  assumed  no  effort  is  necessary 

2.4  Software  Install  &  Test 

2  full-time  vendor  staff  and  2  full-time  GS-14s  for  2  weeks,  travel  costs 
for  the  vendor  staff 

2.5  Data  conversion  and  Entry 

75%  of  1  GS-12  for  17  months,  4  GS-7s  for  5  months  of  data  entry 
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2.6  User  Acceptance 

Assumed  to  be  included  in  element  2.4 

2.7  Training 

Roll-up  element 

2.7.1  Training  Development 

2  GS-14s  full-time  for  1  year,  50%  of  1  GS-12  for  6  months 

2.7.2  Train-the-Trainer 

Assumed  the  trainers  are  experts  who  need  no  additional  training 

2.7.3  End  User  Training 

Assumed  to  be  in  element  3.5 

3.0  Operation  and  Snpport 

Roll-up  element 

3.1  System  operations 

Status  quo  costs  with  phase  out  plan 

3.2  Technical  support 

Status  quo  costs  with  phase  out  plan,  1  GS-12  full-time  after  system 
implementation 

3.3  Software  maintenance  and 
upgrades 

$50,000/yr  for  normal  upgrades 

ACCES  application  maintenance  estimated  using  SEER-SEM. 

Assumed  maintenance  begins  at  the  end  of  the  development  life  cycle. 
100%  of  system  will  be  maintained,  with  an  average  annual  change  rate 
of  5%. 

3.4  Hardware  maintenance  and 
upgrades 

$50,000/yr 

3.5  Recurring  training 

Three  45-hour  training  sessions  per  year  -  GS-14 

3.6  Other  costs 

Status  quo  costs  with  phase  out  plan 
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Alternative  #2,  Searchable  Database  (Constant  FY02  $  000) 


Constant  Years  (000) 

2002 

2003 

2004 

2005 

2006 

Totals 

1.0  System  Acquisition 

$  323.67 

$  1,625.13 

$  214.66 

... 

... 

- 1 

2,163.36 

1.1  Acquisition  Support 

$  B1.25 

... 

... 

... 

... 

$  61,25 

1.1.1  Project  Management 

J  9.73 

... 

$  9.73 

1,1 2  RFP  Preparation 

$  22.08 

J  22.08 

1 ,1 .3  RFP  Evaluation  and  Vendor  Selection 

$  29.44 

i  29.44 

1 .2  Hardware  Investment 

___ 

... 

... 

___ 

... 

1.3  Software  Investment 

$  2B2.33 

$  1,B25,13 

$  214.BB 

$  2,102.12 

1,3.1  Software  Development 

$  136,33 

$  1,625.13 

$  214,66 

$  1,976.12 

1,3.2  SoftwareUcenses 

$  126.00 

i  126.00 

1.4  Infrastructure  Enhancements 

... 

... 

... 

___ 

... 

2.0  System  Implementation 

$  60.42 

$  370.48 

$  171.32 

... 

... 

$  602.22 

2.1  Project  Management 

$  20,68 

$  62,05 

$  10,34 

___ 

$  93.08 

2.2  Contract  Management 

$  6.62 

$  13,25 

$  2.21 

... 

___ 

$  22.08 

2.3  Hardware  Installation  and  Test 

___ 

... 

___ 

2.4  Software  Installation  and  Test 

___ 

$  39.98 

... 

___ 

$  39.98 

2.5  Data  Conversion  and  Entry 

$  33.12 

$  77,31 

$  66.38 

. 

___ 

$  176.80 

2.6  User  Acceptance 

... 

___ 

2.7  Training 

___ 

$  217,87 

$  52.41 

$  270.28 

2.7.1  Training  Development 

_ 

$  217.87 

i  52,41 

... 

... 

$  270.28 

2.7.2  Train-the-Trainer 

_ 

... 

2.7.3  End  User  Training 

... 

... 

3.0  Operations  and  Support 

$  983.70 

$  983.70 

$  1,228.33 

$  1,040.96 

$  734.32 

$  4,971.00 

3.1  Systems  Operations 

$  779,07 

$  779,07 

$  584,30 

$  389.53 

$  194,77 

$  2,726.74 

3.2  Technical  Support 

$  77,91 

$  77,91 

$  132.02 

$  127.26 

$  107,79 

$  522.88 

3.3  Software  Maintenance  and  Upgrades 

$  50,00 

$  50,00 

$  433,91 

$  452.74 

$  367,03 

$  1,353.68 

3.4.  Hardware  Maintenance  and  Upgrades 

$  50,00 

$  50,00 

$  50,00 

$  50,00 

$  50,00 

$  256.00 

3.5  Recurring  Training 

$  8,05 

$  8,05 

$  8,05 

$  24.16 

3.6.  Other  Costs 

$  26,73 

$  26,73 

$  20,04 

$  13,36 

$  6,68 

$  93.54 

Totals 

$  1,367.70 

$  2,979.31 

$  1,614.31 

$  1,040.95 

$  734.32 

$7,736.58 
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Alternative  #2,  Searchable  Database  (Then  Year  $  000) 


Then  Years  fOOO) 

2002 

2003 

2004 

2005 

2006 

Totals 

1.0  System  Acquisition 

$  323.57 

$  1,696.64 

$  228.83 

... 

... 

$  2,249.04 

1.1  Acquisition  Support 

$  B1.25 

... 

... 

... 

... 

$  61.25 

1.1.1  Proiect  Manaaement 

$  9.73 

$  9,73 

1.1,2  RFP  Preparation 

$  22,08 

J  22.08 

1 .1 .3  RFP  Evaluation  and  Vendor  Selection 

$  29.44 

%  29.44 

1.2  Hardware  Investment 

___ 

___ 

___ 

1.3  Software  Investment 

$  2B2.33 

$  1  ,B96.B4 

$  228.83 

... 

... 

$  2,187.79 

1.3.1  Soltware  Development 

$  136.33 

$  1 .696.64 

i  228.83 

... 

$  2,061.79 

1.3.2  Soltware  Licenses 

$  126.00 

... 

%  126.00 

1.4  Infrastructure  Enhancements 

^ 

... 

... 

___ 

2.0  System  Implementation 

$  60.42 

$  386.78 

$  182.63 

... 

... 

$  629.83 

2.1  Project  Management 

$  20.68 

$  64.78 

$  11.02 

— 

— 

$  96.49 

2.2  Contract  Management 

$  6,62 

$  13.83 

$  2,35 

— 

$  22.81 

2.3  Hardware  Installation  and  Test 

... 

... 

— 

2.4  Software  Installation  and  Test 

... 

$  42.B2 

— 

$  42.62 

2.5  Data  Conversion 

$  33.12 

$  80.71 

$  70.7B 

___ 

$  184.59 

2.6  User  Acceptance 

___ 

... 

... 

... 

___ 

2.7  Training 

___ 

$  227.4B 

$  55.8B 

... 

___ 

$  283.33 

2.7.1  Traininq  Development 

_ 

S  227.46 

i  55,86 

*  283.33 

2.7.2  Train-the-Trainer 

2.7.3  End  User  Traininq 

... 

... 

3.0  Operations  and  Support 

$  983.70 

$  1,026.98 

$  1,309.40 

$  1,132.66 

$  816.83 

$  6,268.47 

3.1  Systems  Operations 

$  779.07 

$  813.35 

$  622.86 

$  423.81 

$  216,39 

$  2,855.48 

3.2  Technical  Support 

$  77,91 

$  81.33 

$  140,73 

$  138.46 

$  119.75 

$  558.19 

3.3  Software  Maintenance  and  Upgrades 

$  50.00 

$  52.20 

$  462.55 

$  492.58 

$  407.77 

$  1,465.10 

3.4.  Hardware  Maintenance  and  Upgrades 

$  50.00 

$  52.20 

$  53,30 

$  54.40 

$  55,55 

$  265.45 

3.5  Recurring  Training 

$  8.59 

$  8,76 

$  8.95 

$  26.30 

3.6,  Other  Costs 

$  26,73 

$  27.90 

$  21,37 

$  14.54 

$  7,42 

$  97.95 

Totals 

$  1,367.70 

$  3,110.40 

$  1,720.85 

$  1,132,56  1$  815.83 

$8,147.33 
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Air  Combat  Capability  Enhancement  Suite  (ACCES) 
Cost  Estimating  Methodology  Matrix 
Alternative  3:  Searchable  Database  with  Expert  Interface 


Element 

Estimating  Methodology 

1.0  System  Acquisition 

Roll-up  element 

1 . 1  Acquisition  Support 

Roll-up  element 

1.1.1  Project  Management 

10%  of  1  GS-15’s  labor  for  9  months 

1.1.2  RFP  Preparation 

35%  of  2  GS-12s  labor  for  6  months 

1.1.3  RFP  Evaluation  and 

Vendor  Selection 

100%  of  2  GS-12s  labor  for  3  months 

1.2  Flardware  Investment 

Assumed  all  hardware  requirements  are  in  place 

1.3  Software  Investment 

Roll-up  element 

1.3.1  Software  Development 

Parametric  estimate  using  SEER-SEM  software  cost  estimating  tool. 
Average  loaded  labor  rate  $14,300  per  month. 

Assumed  development  of  an  Oracle  database  sized  between  1,000  and 
2,000  function  points  (40,000  to  80,000  source  lines  of  code) 

Platform:  Internet  Development 

Application  Type:  Object  Oriented  Database 

Acquisition  Method:  New  Development 

Development  Method:  Spiral 

Development  Standard:  Commercial 

Also  assumed  SAS  integration  of  between  100  and  200  function  points 
(3,200  to  6,400  source  lines  of  code). 

Platform:  Internet  Development 

Application  Type:  Business  Analysis  Tool 

Acquisition  Method:  New  Development 

Development  Method:  Off-the-Shelf  Integration 

Development  Standard:  Commercial 

Also  assumed  development  of  expert  interface  tool  consisting  of  5 
templates,  coded  in  a  4*  generation  programming  language.  Each 
template  sized  between  250  and  450  function  points  (5,000  to  9,000 
source  lines  of  code) 

Platform:  Internet  Development 

Application  Type:  Business  Analysis  Tool 

Acquisition  Method:  New  Development 

Development  Method:  Spiral 

Development  Standard:  Commercial 

1.3.2  Software  Licenses 

Oracle  and  SAS  licenses  for  500  users 

1.4  Infrastructure  Enhancements 

Assumed  all  network  requirements  are  in  place 
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2.0  System  Implementation 

Roll-up  element 

2.1  Project  Management 

50%  of  1  GS-14’s  labor  for  20  months 

2.2  Contract  Management 

15%  of  1  GS-12’s  labor  for  22  months  (assuming  that  CM  support  will 
be  required  during  the  RFP  evaluation  period) 

2.3  Hardware  Installation  &  Test 

Pursuant  to  element  1 .2,  assumed  no  effort  is  necessary 

2.4  Software  Install  &  Test 

2  full-time  vendor  staff  and  2  full-time  GS-14s  for  3  weeks,  travel  costs 
for  the  vendor  staff 

2.5  Data  conversion  and  Entry 

67%  of  1  GS-12  for  19  months,  4  GS-7s  for  5  months  of  data  entry 

2.6  User  Acceptance 

Assumed  to  be  included  in  element  2.4 

2.7  Training 

Assumed  no  training  required 

3.0  Operations  and  Snpport 

Roll-up  element 

3.1  System  operations 

Status  quo  costs  with  phase  out  plan 

3.2  Technical  support 

Status  quo  costs  with  phase  out  plan,  1  GS-12  full-time  after  system 
implementation 

3.3  Software  maintenance  and 
upgrades 

$50,000/yr  for  normal  upgrades 

ACCES  application  maintenance  estimated  using  SEER-SEM. 

Assumed  maintenance  begins  at  the  end  of  the  development  life  cycle. 
100%  of  system  will  be  maintained,  with  an  average  annual  change  rate 
of  5%. 

3.4  Hardware  maintenance  and 
upgrades 

$50,000/yr 

3.5  Recurring  training 

Three  45-hour  training  sessions  per  year  -  GS-14 

3.6  Other  costs 

Status  quo  costs  with  phase  out  plan 
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Alternative  #3,  Searchable  Database  with  Expert  Interface  (Constant  Year  FY02  $  000) 


Constant  Year  $(000) 

2002 

2003 

2004 

2005 

2006 

Totals 

1.0  System  Acquisition 

$  721.10 

$  4,123.82 

$  1,227.06 

... 

... 

$  6,071.98 

1.1  Acquisition  Support 

$  8B.01 

_ 

... 

... 

$  86.01 

1.1.1  ProjetS  Management 

$  10.95 

$  10.95 

1.12  RFP  PreparaJon 

$  30.91 

$  30.91 

1 .1 .3  RFP  Evaluation  and  Vendor  Selection 

$  44.15 

$  44.15 

1.2  Hardware  Investment 

... 

... 

... 

1 .3  Software  Investment 

$  635.09 

$  4,123.02 

$  1,227.06 

___ 

$  5,985.97 

1.3.1  Software  Development 

$  509.09 

$  4,123.82 

$  1 ,227,06 

$  5,859.97 

1.32  SoftiAiareUcenses 

$  126.00 

$  126.00 

1.4  Infrastructure  Enhancements 

___ 

... 

... 

2.0  System  Implementation 

$  45.69 

$  134.46 

$  173.84 

_ 

_ 

$  353.99 

2.1  Project  Management 

$  15.51 

$  62.05 

$  25.05 

... 

... 

$  103.42 

2.2  Contract  Management 

$  5.52 

$  13.25 

$  5.52 

... 

t$  24.29 

2.3  Hardware  Installation  and  Test 

... 

... 

... 

... 

2.4  Software  Installation  and  Test 

$  49.63 

... 

... 

$  49.63 

2.5  Data  Conversion 

$  24.85 

$  59.17 

$  92.04 

... 

$  176.66 

2.6  User  Acceptance 

___ 

... 

2.7  Training 

... 

... 

... 

... 

... 

2.7.1  Training  Development 

... 

2.72  Train-the-Trainer 

_ 

2.7.3  End  User  Traininq 

3.0  Operations  and  Support 

$  983.70 

$  983.70 

$  1,959.34 

$  1,902.36 

$  1,195.64 

$  7,024.74 

3.1  Systems  Operations 

$  779.07 

$  779.07 

$  504.30 

$  309.53 

$  194.77 

$  2,726.74 

3.2  Technical  Support 

$  77.91 

$  77.91 

$  109.94 

$  127.26 

$  107.79 

$  500.81 

3.3  Software  Maintenance  and  Upgrades 

$  50.00 

$  50.00 

$  1,195.05 

$  1,322.20 

$  836.41 

$  3,453.66 

3.4.  Hardware  Maintenance  and  Upgrades 

$  50.00 

$  50.00 

$  50.00 

$  50.00 

$  50.00 

250.00 

3.5  Recurring  Training 

... 

... 

___ 

3.6.  Other  Costs 

$  26.73 

$  26.73 

$  20.04 

$  13.36 

$  6.60 

$  93.54 

Totals 

$  1,750.49 

$  5,241.98 

$  3,360.25 

$  1,902.36 

$  1,195.64 

$13,450.72 

Alternative  #3,  Searchable  Database  with  Expert  Interface  (Then  Year  $  000) 


Then  Year  $(000) 

2002 

2003 

2004 

2005 

2006 

Totals 

1.0  System  Acquisition 

$  721.10 

$  4,305.26 

$  1,308.05 

... 

... 

$  6,334.42 

1.1  Acquisition  Support 

$  86.01 

... 

... 

... 

... 

$  86.01 

1.1.1  Project  Management 

$  10.95 

$  10.95 

1.1 .2  RFP  Preparation 

$  30.91 

$  30.91 

1 .1 .3  RFP  Evaluation  and  Vendor  Selection 

$  44.15 

$  44.15 

1.2  Hardware  Investment 

... 

... 

... 

... 

... 

... 

1.3  Software  Investment 

$  635.09 

$  4,305,26 

$  1.308,05 

___ 

$  6,248.41 

1.3.1  Software  Development 

$  509,09 

$  4,305.26 

$  1 ,308,05 

$  6.122.41 

1.32  SoftwareUcenses 

$  126.00 

$  126.00 

1 ,4  Infrastructure  Enhancements 

___ 

2.0  System  Implementotion 

$  45.69 

$  140,38 

$  185.32 

_ 

_ 

$  371.38 

2.1  Project  Management 

$  15.51 

$  64.79 

$  27.56 

... 

-J 

107.85 

2.2  Contract  Management 

$  5.52 

$  13.83 

$  5.00 

... 

$  25.23 

2.3  Hardware  Installation  and  Test 

... 

... 

... 

... 

... 

2.4  Software  Installation  and  Test 

$  52.91 

___ 

$  52.91 

2.5  Data  Conversion 

$  24.85 

$  61.77 

$  99.96 

... 

... 

$  105.39 

2.6  User  Acceptance 

... 

... 

... 

... 

... 

... 

2.7  Training 

... 

... 

2.7.1  Training  Development 

2.7.2  Train-the-Trainer 

_ 

2-7.3  End  User  Traininq 

_ 

3.0  Operations  and  Support 

$  983.70 

$  1,026.98 

$  2,088.66 

$  2,069.77 

$  1,328.36 

$  7,497.46 

3.1  Systems  Operations 

$  779.07 

$  813.35 

$  622.86 

$  423.81 

$  216.39 

$  2,855.49 

3.2  Technical  Support 

$  77.91 

$  81.33 

$  117.20 

$  13B.4B 

$  119.75 

534.65 

3.3  Software  Maintenance  and  Upgrades 

$  50.00 

$  52.20 

$  1,270.93 

$  1,430.55 

$  929.25^ 

$  3,743.93 

3.4.  Hardware  Maintenance  and  Upgrades 

$  50.00 

$  52.20 

$  53.30 

$  54.46 

$  55.55 

$  265.45 

3.5  Recurring  Training 

... 

... 

... 

... 

... 

... 

3.6.  Other  Costs 

$  28.73 

$  27.90 

$  21.37 

$  14.54 

$  7.42 

$  97.95 

Totals 

$  1,750.49 

$  5,472.63 

$  3,582.03 

$  2,069.77 

$  1,328.36 

$14,203.26 
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Project  Schedule 

Alternative  #2,  Searchable  Database 


ID 

Task  Name 

Start 

Finish 

1 

Acquisition  Support 

Mon  10/1/01 

Fri  5/31/01 

2 

Project  Management 

Mon  10/1/01 

Fri  5/31/01 

3 

RFP  Preparation 

Mon  10/1/01 

Fri  3/29/02 

4 

RFP  Evaluation  and  Vendor  Selection 

Mon  4/1/02 

Fri  5/31/02 

5 

System  Implementation 

Mon  4/1/02 

Fri  1/30/04 

6 

Program  Management 

Mon  6/3/02 

Fri  11/28/03 

7 

Contract  Management 

Mon  4/1/02 

Fri  11/28/03 

8 

Software  Development 

Mon  6/3/02 

Fri  11/28/03 

9 

Software  Installation  and  Testing 

Mon  12/1/03 

Fri  12/12/03 

10 

Data  Conversion  and  Entry 

Mon  4/1/02 

Fri  1/30/04 

11 

Training 

Mon  12/2/02 

Fri  11/28/03 

12 

Training  Development 

Mon  12/2/02 

Fri  11/28/03 

2001 


2002 


2003 


2004 
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Project  Schedule 

Alternative  #3,  Searchable  Database  with  Expert  Interface 


ID 

Task  Name 

Start 

Finish 

1 

Acquisition  Support 

Mon  10/1/01 

Fri  6/28/01 

2 

Project  Management 

Mon  10/1/01 

Fri  6/28/01 

3 

RFP  Preparation 

Mon  10/1/01 

Fri  3/29/02 

4 

RFP  Evaluation  and  Vendor  Selection 

Mon  4/1/02 

Fri  6/28/02 

5 

System  Implementation 

Mon  4/1/02 

Fri  4/30/04 

6 

Program  Management 

Mon  7/1/02 

Fri  2/27/04 

7 

Contract  Management 

Mon  4/1/02 

Fri  2/27/04 

8 

Software  Development 

Mon  7/1/02 

Fri  2/27/04 

9 

Software  Installation  and  Testing 

Mon  3/1/03 

Fri  3/19/04 

10 

Data  Conversion  and  Entry 

Wed  5/1/02 

Fri  1/30/04 

2001 


2002 


2003 


2004 


fiSSSHSSSSSS 
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APPENDIX  B.  PAIR-WISE  COMPARISON  OF  ALTERNATIVES 


Table  B-1  shows  the  all  the  benefit  pairwise  comparisons. 


Table  B-I,  Global  Table  of  Pairwise  Comparisons 


ACCESS 

9 

8 

7 

6 

5 
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2 

I 
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8 

9 
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9 

8 

7 
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2 

I 

2 
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4 

5 

6 

7 

8 

9 

EFFICIEN 

ACCESS 

9 

8 

7 

6 

5 

4 

3 

2 

I 

2 

3 

4 

5 

® 

7 

8 

9 

SCOPE 

ACCESS 

9 

8 

7 

6 

5 

4 

3 

2 

I 
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3 

4 

5 

6 

7 

8 

9 

USABE'TY 
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9 

8 

(7) 

6 

5 

4 

3 

2 

I 
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3 

4 

5 

6 

7 

8 

9 

EFFICIEN 
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9 

8 

7 
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4 

3 

2 

I 

2 

3 

4 

5 

6 

7 

8 

9 

SCOPE 

ACCURACY 

9 

8 

@ 

6 

5 

4 

3 

2 

I 
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4 

5 

6 

7 

8 

9 
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8 

7 
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5 

4 

3 

2 

I 

2 

3 

4 

5 

7 

8 

9 

SCOPE 

EFFICIEN 

9 

8 

7 

6 

5 

4 

3 

2 

I 

(D 

3 

4 

5 

6 

7 

8 

9 

USABE'TY 

SCOPE 

9 

8 

6 

5 

4 

3 

2 

I 

2 

3 

4 

5 

6 

7 

8 

9 

USABE'TY 

Ratings:  1  =  Equal;  3  =  Moderate;  5  =  Strong;  7  =  Very  Strong;  9  =  Extreme 

Example  of  interpretation:  Accuracy  (ACCURACY)  is  strongly  more  important  than  scope 

(SCOPE). 


While  Table  B-1  shows  the  global  summary  of  comparisons,  the  purpose  of  this 
evaluation  was  to  assess  potential  benefits  of  fielding  new  anthropometric  information  systems 
compared  to  the  status  quo.  It  was  also  to  “quantify”  the  subjective  qualities  of  both  the  existing 
and  presented  system  benefits.  When  combined  with  cost  estimate,  this  data  will  contribute  to  the 
overall  CBA  results. 

Please  note  that  this  appendix  is  intended  to  support  information  found  in  the  benefits 
analysis.  Section  2.6,  it  may  repeat  information  found  in  the  body  of  this  document.  The 
repetition  is  intended  to  allow  this  appendix  to  stand  alone  as  a  separate  entity. 

Recall  that  the  customer  defined  goal  for  the  three  alternative  anthropometric  systems 
was  to  develop  an  international  anthropometric  size,  shape,  fit  and  accommodation  information 
system  for  efficient  and  effective  use  by  designers,  manufacturers,  buyers,  and  decision  makers 
across  a  number  of  domains.  It  is  with  respect  to  this  goal  that  all  alternatives  were  assessed. 

A  table  of  global  benefits  can  be  found  in  Table  B-2  below. 


Table  B-2.  Benefits  and  Their  Definitions 


Benefit 

Working  Definition 

Accessibility 

System  impact  on  ease  of  accessing  anthropometric  information;  the 
number  and  type  of  individuals  that  can  locate  the  info. 

Accuracy 

System  impact  on  currency,  correctness,  and  direct  use  of 
fit/accommodation  information.  Information  provided  is  the  correct 
solution  for  the  task  at  hand. 

Efficiency 

System  impact  on  speed  of  finding  anthropometric  fit/ accommodation 
information.  A  more  efficient  system  will  require  less  time  invested. 

Scope 

System  impact  on  sufficiency  of  anthropometric  fit/accommodation 
information  provided  to  user.  Information  provided  is  in-depth  enough  to 
meet  user's  needs. 

Usability 

Ease  of  system  use  in  accessing  anthropometric  fit/accommodation 
information.  Eevel  of  effort  required  of  designer  to  find  information.  A 
highly  useable  system  requires  little  effort. 
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To  reiterate,  the  investment  alternatives  were: 


•  Alternative  #1  Status  Quo  -  The  first  alternative  assessed  by  the  Expert  Choice  Team  was 
the  fielded  system  described  in  Section  2.1.2. 

•  Alternative  #2  Searchable  Database  -  The  second  alternative  discussed  by  the  Expert 
Choice  Team  was  the  searchable  database  described  in  Section  2.3.2. 

•  Alternative  #3  Searchable  Database  with  Expert  Interface  -  The  final  alternative  assessed 
by  the  Expert  Choice  Team  was  the  searchable  database  combined  with  an  expert 
interface  capability,  described  in  Section  2.3.2. 

When  ranking  the  three  alternatives  for  each  benefit,  the  Expert  Choice  team  followed  specific 
steps.  These  were: 

•  Review  goal,  benefits,  and  alternative  definitions, 

•  Discuss  examples  of  each  benefit,  then  rank  all  benefits,  and 

•  Identify  variations  between  alternatives,  then  rank  alternatives  for  each  benefit 

The  following  sections  outline  the  pairwise  comparisons  made  with  respect  to  the 
investment  alternatives  and  the  benefits. 


ACCESSIBILITY 

Table  B-3  shows  the  pairwise  comparisons  associated  with  the  Accessibility  benefit. 


Table  B-3.  Accessibility  Comparison 


SEAT  QUO 

9 

8 

7 
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5 
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2 
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5 
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(2) 

8 

9 

DAT'BASE 
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9 

8 

7 

6 

5 

4 

3 

2 

1 

2 

3 

4 

5 

6 

7 

9 
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9 

8 

7 

6 

5 

4 

3 

2 

1 

2 
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4 

5 

6 

7 

8 

9 

DBW/EXP 

Definition 

System  impact  on  ease  of  accessing  anthropometric  information;  the  number  and  type  of 
individuals  that  can  locate  the  info. 

Example 

Some  designers  have  access  to  up  to  date  information  or  know  where  to  look  and  some 
do  not.  Depending  on  the  publicity  of  the  source,  some  information  is  simply  difficult  to  find  and 
asking  an  expert  is  the  only  way  designers  would  even  know  it  existed. 

Answers  the  question,  "Who  knows  about  and  has  access  to  the  required  information?" 

Background  on  Investment  Alternatives 

“The  lack  of  readily  available  and  reliable  data  on  (anthropometry). . .  is  ...  a  problem.” 

(Ergotech,  2001).  It  is  known  that  ANSUR  88  is  reasonably  available  for  use,  but  its  current 
utility  may  be  questionable.  On  the  other  hand,  CAESAR  data  is  more  up-to-date  and  accurate, 
but  few  decision  makers  have  access  to  it. 
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Alternative  #1 

With  this  alternative,  novices  would  seek  out  information  on  their  own  through  books, 
journals,  and  the  like.  Information  could  be  gathered  from  printed  standards  and  handbooks  such 
as  MIL-STD-1472,  and  MIL-HDBK-743,  as  well  as  journal  articles,  and  human  factors 
textbooks.  The  existence  of  the  appropriate  information  and  the  expertise  needed  to  find  or  use 
that  information  may  not  be  known  by  the  user.  An  expert  consultant,  such  as  someone  at 
AFRL/HE,  is  sometimes  used  to  find  the  appropriate  data  and  put  it  in  a  useful  form  for  the 
particular  project  or  problem.  For  example,  a  manufacturer  of  an  oxygen  mask  may  not  know 
there  is  a  3-D  face  data  resource  available. 

Alternative  #2 

All  of  the  above  accessibility  is  included.  Previously  unincorporated  sources  will  also  be 
brought  together,  putting  all  pertinent  information  in  the  same  place.  As  a  result,  information  is 
located  in  or  its  location  can  be  identified  from  one  central  information  system  location.  This 
would  include  data  from  outside  organizations  worldwide. 

Alternative  #3 

All  of  the  above  accessibility  is  included. 


ACCURACY 


Table  B-4  shows  the  pairwise  comparisons  associated  with  the  accuracy  benefit. 

Table  B-4.  Accuracy  Comparison 
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Definition 

System  impact  on  currency,  correctness,  and  direct  use  of  fit/accommodation 
information.  Information  provided  is  the  correct  solution  for  the  task  at  hand. 

Example 

Information  provided  is  up  to  date  and  intended  for  design  use.  Information  is 
appropriate  for  the  context  and  intended  use. 

Answers  question,  "Flow  correct  is  the  information  provided  as  it  pertains  to  its  intended 

use?" 

Background  on  Investment  Alternatives 

Percentiles  and  averages  are  not  always  the  best  or  most  accurate  source  of  design 
information.  Similarly,  current  3-D  models  are  poor  representations  based  on  1-D  measurements 
(Robinette,  2001).  “The  lack  of  readily  available  and  reliable  data  on  (anthropometry). . .  and 
infrequent  updating  of  populations  is  also  a  problem”  (Ergotech,  2001).  In  fact,  an  estimated  9 
percent  of  all  aircraft  accidents  are  anthropometry-related  (Zehner,  2001).  Furthermore, 
“Furniture,  work  tools,  and  personal  protective  equipment  are  areas  where  inadequate 
anthropometric  data  can  result  in  ill-fitting  and  potentially  unsafe  designs. . .”  (Phillips,  1995,  p. 

1) 
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Alternative  #1 

Novices  must  use  printed  texts  with  summary  statistics  from  preset  surveys  such  as  mean, 
standard  deviation,  and  percentiles.  The  sample  used  may  not  be  representative  of  the  desired 
population.  Even  the  experts  face  the  same  problems,  but  know  of  more/better  resources  for 
accurate  information. 

Alternative  #2 

The  information  used  by  experts  should  be  accurate  and  up  to  date.  The  database  can  be 
added  by  the  user  and  updated  using  a  standard  database  format.  Novices  using  this  system  could 
get  accurate  information,  but  may  not  ask  the  right  questions 

Alternative  #3 

This  system  can  be  used  by  experts  for  advanced  problems,  research,  or  to  override  the 
generic  output  if  desired.  This  might  be  necessary  to  use  the  most  up-to-date  methods  as 
technology  advances  are  made.  The  systems  should  also  have  about  the  same  high  level  of 
accuracy  for  novices. 


EFFICIENCY 


Table  B-5  shows  the  pairwise  comparisons  associated  with  the  efficiency  benefit. 

Table  B-5.  Efficiency  Comparison 
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Definition 

System  impact  on  speed  of  finding  anthropometric  fit/accommodation  information.  A 
more  efficient  system  will  require  less  time  invested. 

Example 

A  database  can  give  immediate  results,  while  contacting  an  expert  could  take  one  or  more 
weeks  to  receive  answer. 

Answers  question,  "How  long  does  it  take  to  get  answer?" 


Background  on  Investment  Alternatives 

A  70  percent  solution  on  time  is  more  favorable  than  a  95  percent  solution  too  late  to  be 

of  use. 


Alternative  #I 

Novices  using  books,  journals,  etc.,  must  look  up  solutions  on  their  own.  For  novices 
asking  questions,  it  could  take  some  time  to  get  response.  For  experts  answering  questions,  this 
method  is  not  efficient  as  information  may  or  may  not  be  readily  available.  The  consultant  may 
or  may  not  have  the  software  and  data  resources  readily  available  to  accomplish  this;  therefore, 
these  may  have  to  be  separately  acquired.  The  information  is  not  in  a  central  location,  nor  can  its 
location  be  found  from  one  central  resource. 


B-4 


Alternative  #2 

There  was  improved  efficiency  for  experts  using  this  system.  Information  will  be  located 
in  the  database  or  its  location  can  be  identified  from  one  central  information  system,  improving 
efficiency.  This  would  include  data  from  outside  organizations  worldwide.  Database  would  be 
one-  and  three-dimensional  searchable  on-line,  with  sample  merging,  weighting  and  segmenting 
capability.  This  enables  the  creation  of  a  more  representative  sample  of  the  population  of 
interest,  such  as  the  Joint  Primary  Aircrew  Training  System  (JPATS)  sample,  or  the  National 
Health  and  Nutrition  Examination  Survey  (NHANES)  sample. 

This  solution  does  little  to  help  novice,  however. 

Alternative  #3 

Expert  interface  should  improve  efficiency  by  significantly  reducing  time  required  to 
identify  and  locate  necessary  information.  Should  have  about  same,  improved,  efficiency  for 
novice. 


SCOPE 


Table  B-6  shows  the  pairwise  comparisons  associated  with  the  scope  benefit. 


Table  B-6.  Scope  Comparison 
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Definition 

System  impact  on  completeness  and  sufficiency  of  anthropometric  fit/accommodation 
information  provided  to  user.  Information  provided  is  in-depth  enough  to  meet  user's  needs. 

Example 

A  book  could  provide  quick  anthropometric  information,  but  not  enough  of  the  right  kind 
of  information  needed.  A  database  could  provide  the  right  kind  of  information,  but  too  much  of  it 
for  the  user  to  comprehend. 

Answers  the  question,  "What  do  I  need  to  know  to  make  an  anthro/design  decision?" 

Background  on  Investment  Alternatives 

Matching  crew  with  equipment  is  a  two-part  process  (Robinette,  1999).  In-depth 
information  is  needed  for  both  steps: 

1.  Assigning  crew  to  appropriate  billets  (over  time),  and 

2.  Sizing,  issuing  and  stocking,  appropriate  apparel  and  personal  equipment. 

Alternative  #1 

A  novice  searching  for  information  may  not  get  all  the  information  that  he/she  needs. 
Expert  searching  for  information  might  be  able  to  find  everything  he/she  needs,  for  example: 
One-dimensional  (traditional  tape  measure,  caliper  type)  data  by  survey  in  spreadsheets  on  CD, 
such  as  the  Army  1988  survey,  the  USAF  mini-survey,  etc. 

Alternative  #2 
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This  system  provides  improved  scope  for  experts  and  novices.  Lessons  learned  and  fit 
and  accommodation  maps  are  included.  Also  included  are  Human  system  equipment  size  and 
shape  information  and  interfacing  3-D  visualization  and  statistical  analysis  tools.  As  a  result, 
novices  may  have  difficulty  identifying  and  using  appropriate  information  (see  Usability) 

Alternative  #3 

This  alternative  has  the  same  scope  as  above. 


USABILITY 

Table  B-7  shows  the  pairwise  comparisons  associated  with  the  usability  benefit. 


Table  B-7.  Usability  Comparison 
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Definition 

Ease  of  system  use  in  accessing  anthropometric  fit/accommodation  information.  Level  of 
effort  required  of  designer  to  find  information.  A  highly  useable  system  requires  little  effort. 

Example 

A  highly  useable  system  provides  information  in  a  simple  format  with  low  demands  on 
user.  There  is  little  training  or  extra  work  involved  with  a  highly  useable  system. 

Answers  question,  "How  easy  is  it  to  access  the  required  information?" 

Background  on  Investment  Alternatives 

Current  3-D  anthropometry  information  is  not  in  a  readily  usable  form  and  distilling  is 
necessary  (Robinette,  1999).  “The  future  application  of  this  (anthropometric)  knowledge  is  in 
software  packages  and  their  ability  to  interact  with  one  another.”  (Tucker  &  Brattin,  2000,  p.  5) 

Alternative  #I 

For  novices  asking  questions,  this  system  is  reasonably  usable.  They  simply  have  to  ask 
questions  of  experts  and  wait  for  answer.  They  also  can  just  look  in  books  and  find  answers. 

For  experts  answering  questions,  this  system  is  much  less  usable.  This  is  because 
information  is  not  very  accessible. 

Alternative  #2 

This  alternative  is  moderately  usable  by  experts.  Electronic  accommodation  or  fit 
prediction  models  are  included.  On  the  other  hand,  this  system  is  not  very  usable  by  novices.  It 
requires  a  knowledgeable  expert  to  locate  the  most  appropriate  information  and  answer  the 
questions  in  many  cases. 

Alternative  #3 

This  system  has  an  interface  that  allows  experts  and  novices  to  locate  the  appropriate 
information  in  the  form  needed,  by  simply  asking  a  question.  It  will  walk  them  through  to  the 
answer.  Some  sample  questions  include: 
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•  Would  this  candidate  be  able  to  safely  fly  USAF  fighter  aircraft?  Answer  would  include 
the  aircraft  that  could  be  safely  flown  as  well  as  any  areas  that  might  be  on  the  borderline 
and  by  how  much. 

•  What  percentage  of  the  population  would  be  able  to  fly  this  aircraft  if  this  change  were 
made?  The  interface  would  ask  the  novice  questions  to  ascertain  exactly  what  the 
population  of  interest  might  be.  The  answer  would  include  the  overall  population,  the 
male  and  the  female  populations  and  the  minority  population  percentages. 

•  What  should  the  requirements  be  for  a  new  oxygen  mask?  The  interface  would  ask  the 
novice  questions  to  ascertain  how  the  oxygen  mask  is  intended  to  fit  and  function,  and 
who  the  intended  users  would  be  using  examples  and  pictures  (3-D  if  available)  from  the 
fit  history  files.  The  answer  would  include  the  statements  to  be  written  in  the 
specification,  and  the  data  on  the  cases  (3-D  if  necessary  and  available)  to  use  to 
accommodate  the  population. 
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APPENDIX  C.  LITERATURE  SEARCH  STRATEGY 


Air  Combat  Capability  Enhancement  Suite  (ACCES) 

Cost  Benefit  Analysis 

Literature  Search  Strategy 

For:  Air  Force  Research  Laboratory 

Wright-Patterson  Air  Force  Base,  OF! 

Background: 

Fluman  Systems  lAC  is  conducting  a  cost  benefit  analysis  on  the  Air  Combat  Capability 
Enhancement  Suite  (ACCES).  This  effort  is  tasked  with  developing  and  demonstrating  an 
anthropometric  fit  and  accommodation  information  system  for  use  by  decision-makers  across  a 
number  of  domains.  It  will  provide  three  major  resources:  a)  a  data  base  of  current  forces  and  the 
best  equipment  size,  adjustments,  etc.  for  usability,  fit,  and  safety,  b)  the  best  mix  of 
stocked/special  order/custom  personal  equipment  sizes  for  rapid  and  cost-effective  deployment, 
and  c)  information  on  areas  of  improvement  for  more  cost-effective  fit,  assignment  entry  and 
fitness  standards,  and  purchasing. 

The  specific  goal  of  this  literature  search  is  to  review  recent  literature  on  the  three  main 
topics:  a)  Human  variability  of  advanced  methods  of  characterizing  human  variability  ,  b) 
anthropometric  variability  and  the  relationship  between  anthropometry  and  equipment  systems, 
and  c)  anthropometry  and  accommodation  data  resources  and  issues,  such  as  new  information 
system  methods,  3-D  shape  searching  that  allow  a  non-professional  to  get  useable  information.. 
The  search  results  will  help  Human  Systems  lAC  identify  and  understand  the  problems  associated 
with  the  above  issues.  This  review  should  also  help  identify  the  available  alternatives  and  what 
the  potential  payoffs  are  for  continuing  R&D  in  the  area  of  anthropometric  databases  and 
information  presentation.  This  information  will  then  be  evaluated  with  the  cost  data  to  determine 
the  alternative  technology  with  the  greatest  return  on  investment. 

The  results  of  the  literature  search  strategy  will  be  used  to  derive  cost  and  benefits  as 
appropriate.  The  results  are  especially  important  in  capturing  the  "value"  of  benefits  in  order  to 
quantify  them  in  our  final  analysis. 


Search  Terms: 

See  attaehed  table  of  terms. 


Key  Authors: 

Claire  Gordon 
Kenneth  W.  Kennedy 
Kathleen  M.  Robinette 
Gregory  F.  Zehner 
Jennifer  Crawford 
Joseph  Eicina 
Jeff  Hudson 
Brian  Corner 
Seven  Daquette 
Eric  Paquet 


Michael  Vannier 
Bruce  Bradtmiller 
Regis  Mollard 
Hein  Daanen 
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Possible  Databases:  (Final  list  to  be  determined  based  on  customer  and  expert  searcher 
recommendations) 

Aerospace  Database 
ISI  Science  Citation  Index 
NASA  Recon 
NTIS 

PsychINFO 

Naval  Research  Labs 

U.S.  Army  Research  Labs 


Example  Articles: 

Topic  1:  Human  Variability  is  More  Than  a  Small  Female  and  a  Large  Male 

Daniels,  G.S..  1952.  The  Average  Man? .  TN-WCRD  53-7.  (AD  10  203)  Wright  Air  Development 
Center,  Wright-Patterson  Air  Force  Base,  OH. 

Hendy,  K.C..  (1990).  Aircrew/Cockpit  compatibility-a  multivariate  problem  seeking  a  multivariate 
solution,  'mAGARD,  Recruiting,  Selection,  Training  and  Military  Operations  of  Female 
Aircrew.  Defense  and  Civil  Institute  of  Environmental  Medician,  Downsview  Ontario. 

Robinette,  Kathleen  M.  and  John  T.  McConville,  1982.  "An  Alternative  to  Percentile  Models",  SAE 
Technical  Paper  810217,  in  1981  SAE  Transactions,  pp.  938-946,  Society  of  Automotive 
Engineers,  Warrendale,  PA. 

Zehner,  G.F.,  Meindl,  R.S.,  and  Hudson,  J.A.  (1993).  A  Multivariate  Anthropometric  Method  for 
Crew  Stations  Design:  Abridged,  AE-TR- 1 992-0 1 64,  Armstrong  Eaboratory,  Air  Force 
Systems  Command,  Wright  Patterson  Air  Force  Base,  OH 

Topic  2:  Need  to  Know  Both  Anthropometric  Variability  AND  The  Relationship  Between 

Anthropometry  and  Equipment  Systems 

Robinette,  K.M.  (2000)  3-D  Fit  Mapping.  In  Proceedings  of  the  International  Ergonomics 

Association  XlVth  Triennial  Congress  and  Human  Factors  and  Ergonomics  Society-44th 
Annual  Meeting,  Human  Factors  and  Ergonomics  Society  Inc.,  Santa  Monica,  CA  . 

Whitestone,  J.J.  and  Robinette,  K.M.  (1997)  Fitting  to  maximize  performance  of  HMD  systems,  in 
Head  Mounted  Displays,  Designing  for  the  User,  editors  Melzer,  J.  and  Moffitt,  K.,  chapter 
7,pp.  175-202,  McGraw  Hill  Publishing,  New  York,  New  York. 

Zehner,  Gregory;  Kennedy,  Kenneth;  Hudson,  Jeffrey;  Ivey,  Earry;  Andrews,  Jenny  Et.  (1997), 

Anthropometric  Accommodation  in  Training  Aircraft,  Proceedings  of  the  Thirty  Fifth  Annual 
Symposium  SAFE  Association,  pp  373-379. 

Topic  3:  Anthropometry  and  Accommodation  Data  Resources  and  Issues 

Churchill,  E.,  T.  Churchill,  and  P.  Kikta.  (1977).  The  AMRL  Anthropometric  Data  Bank  Library: 
Volumes  1-V,  AMRE-TR-77-1  (AD  Ao47  314),  Aerospace  Medical  Research  Eaboratory, 

Air  Force  Systems  Command,  Wright-Patterson  Air  Force  Base,  OH. 
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Coblentz,  A.,  J.C.  Pineau,  and  G.  Ignazi.(1992).  Ergodata  an  on  line  data  base  for  ergonomics, 

Proceedings  of  the  2nd  Pan  Pacific  Conference  on  Occupational  Ergonomics,  "Ergonomics 
in  Occupational  Safety  and  Health  ",  Safety  and  Environmental  Protection  Research  Institute, 
MMl,  Wuhan,  China. 

Robinette,  K.M.,  Vannier,  M.W.,  Rioux,  M.,  and  Jones,  P.R.M.  (1997^  3-D  Surface  Anthropometry: 
Review  of  Technologies,  AGARD  Advisory  Report  No.  329,  Advisory  Group  for  Aerospace 
Research  and  Development,  7  Rue  Ancelle,  92200  Neuilly-Sur- Seine,  France. 

Robinson,  J.,  Robinette,  K.M.  and  Zehner,  G.F.  (1992).  User's  Guide  to  the  Anthropometric  Data 
Base  at  the  Computerized  Anthropometric  Research  and  Design  (CARD)  Laboratory,  AE- 
TR- 1992-0036,  Crew  Systems  Directorate,  Human  Engineering  Division,  Armstrong 
Eaboratory,  Wright-Patterson  Air  Force  Base  OH. 

Rogers-Adams,  Beth  M.,  Andrew,  J.  Capt.,  Zehner,  Gregory  F.  (1998).  Anthropometric 

Accommodation  in  USAF  Training  Aircraft:  A  Comparison  of  Operational  Requirements, 
Proceedings  of  the  Thirty  Sixth  Annual  Symposium  SAFE  Association  pp  250-257. 
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ACCES  CBA  SEARCH  STRATEGY 

Main  Ideas 

Secondary  Terms 

Search  Focus 

Additional  Items  of  Interest 

] 

Anthropometric 

A 

Human  body  size 

A 

Multivariate 

A 

accommodation 

N 

Human  models 

N 

Analysis 

N 

Anthropometry 

D 

Sizes 

D 

Percentile 

D 

Goal:  To  acquire  documents  that  c 

efme  human  variability. 

7 

Anthropometric 

A 

Protective  equipment 

A 

Helmets 

A 

Usability 

accommodation 

N 

Life  support 

N 

G-suit 

N 

Safety 

Cockpit  accommodation 

D 

Cockpit 

D 

Survival  vest 

D 

Anthropometric  survey 

Supplies 

Clothing 

Sizing 

Dimensions 

Fit 

Goal:  To  find  documents  that  exhi 

3it  the  relationship  between  anthropometric  variability  and  equipment  systems. 

3 

Database 

A 

Anthropometric 

A 

Univariate 

A 

Analysis 

3-D  Body  data 

N 

accommodation 

N 

Multivariate 

N 

Anthropometry 

D 

Data  management 

D 

Integrate 

Ergodata 

NACSET 

D 

(Navel  Advanced  Crew  Station 
Evaluation  Technique) 

Goal:  To  find  evaluations  of  anthropometry  and  accommodation  data  resources  and  issues. 

A 

1  and  2  embedded  on/in  3 

A 

Stocked  equipment 

A 

Anthropometric  databases 

A 

N 

Supplies 

N 

Equipment  databases 

N 

D 

Equipment  variability 

Sizes 

Cost  effective 

D 

D 

Goal:  To  determine  what  type  of,  if  any,  anthropometric  and  equipment  databases  exist,  and  whether  it  is  effective  to  combine  anthropometric  fit 

with  accommodation  systems. 
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